FOP2 Stops responding after awhile

nicolas

Hi,

Start fop2 with debug enabled under a gnu screen session:

screen -dmS fop2 /usr/local/fop2/fop2_server -X 511

(you will need to stop the service first "service fop2 stop")

When you notice the problem, attach to the screen session with:

screen -r fop2

And look at the output.... that will give us a hint on what is going on.

Best regards,

tvittito

Thanks for the quick response! I restarted fop2 with debugging this morning as you suggested. And just now the problem popped up again. No changes on the FOP when a call is made. If an extension is is use I can refresh the FOP screen and it will mark the extension as in use, but when the extension is hung up state of that extension remains "in use."

The following is what is currently up. Is there any way to go back further in the log? Or do you notice anything from that?

192.168.12.68   <= <msg data="1|ping||" />

-- PROCESS_FLASH_COMMAND origen 1 accion ping destino 

-- PROCESS_FLASH_COMMAND password 

192.168.12.68   => { 'btn': '0', 'cmd': 'pong', 'data': '0', 'slot': '' }

** MAIN AMI event received...
** MAIN Processing command received from flash clients...

192.168.12.68   <= <msg data="1|ping||" />

-- PROCESS_FLASH_COMMAND origen 1 accion ping destino 

-- PROCESS_FLASH_COMMAND password 

192.168.12.68   => { 'btn': '0', 'cmd': 'pong', 'data': '0', 'slot': '' }

** MAIN End of block from localhost

127.0.0.1       <- Event: Registry
127.0.0.1       <- Privilege: system,all
127.0.0.1       <- ChannelType: SIP
127.0.0.1       <- Domain: sip03.voicemeup.com
127.0.0.1       <- Status: Registered
127.0.0.1       <- Server: 0

** MAIN AMI event received...
** MAIN There's an 'Event' in the event block
** MAIN Event detected block_count = 1 (Event: Registry)
** MAIN There are 1 blocks for processing
** MAIN Answer block cleared

** DIGEST_EVENT: start

** PROCESA_BLOQUE  0
** PROCESA_BLOQUE REGISTRY 0

Thanks Again,

Tyler

nicolas

Hi,

Well, I do not see anything that can help. What I wanted to know if the AMI-Fop2 server connection was still up when you notice the problem. If it is not up, you will see some connection attempts logged, and you should not see anything like:

1

27.0.0.1       <- Event: Registry
127.0.0.1       <- Privilege: system,all

Those are actual AMI events that are being read/received by fop2 server. If the AMI connection stalls or freezes or dies, then fop2 will stop receiving status updates. But for what I see, if those are the latest lines in the screen session, then that did not happen, you might want to look for individual hangup events for the call you saw as frozen...

tvittito

I'll keep investigating. I run on Ubuntu and it looks like it may just be happening on my machine for some reason. Seems to behaving correctly on a XP ran in virtual box. I'm downloading Firefox to the VM right now to see if it's an issue with firefox or ubuntu. I'll keep you advised of what I find.

tvittito

I'm sure I'm just missing something small here but when I bring up "screen -r fop2", how do I scroll?

nicolas

scroll in gnu screen is complicated:

ctrl-a ]

and then the arrows, if I recall correctly.

gregc

Did you find out any workaround for this?

I have the exact same issue with my Firefox/Ubuntu system.

XP virtual works fine though.

tvittito

I'm sorry to hear you are experiencing the same thing, but glad to see that I'm not the only one (I thought I was going crazy). I actually went back to 2.11. I've tried so many things since then I'm not sure if that was the fix or if it was something else. I'm thinking about trying 2.20 again soon. I've just been too busy to think about it at the moment. If I remember correctly I was still having an issue where the page would kind of lock up and not show current status. My very crude fix to this was to force the page to refresh every 15 minutes.

To do that you need to backup this file /var/www/html/fop2/index.html and then edit the following line in it:

Replace

<meta http-equiv="content-type" content="text/html; charset=utf-8" />

With

<meta http-equiv="refresh" content="900" />

I don't know if you need the rest of the meta line, but I made this change about 2 months ago and it's been working fine.

gregc

Mine is instantaneous. That is, I can start up the panel, see a current phone call yet if that phone call hangs up, it doesn't show on the fop2 panel. It still shows connected until I do a page refresh.

I keep an XP virtual open all day anyways and I'm the only one using Ubuntu, so I'll just live with it. Though it would be nice if it could be addressed.

tvittito

You might give chrome a try. It seemed to behave a little bit better in that for some reason. What asterisk version are you running?

nicolas

I use ubuntu/firefox, and I do not experience the issue at all. In order to fix bugs, the idea is for you to capture at least some debug output as explained in this and several other posts. It seems you have a problem with your client if it works for other machines in the system (client is firefox/flash). You might want to test at least with google chrome, that uses html5 websockets instead of flash. Or you can install firebug and open the javascript console and send me some of the output you see there in order to troubleshot. Best regards,

gregc

My XP virtual machine had a firefox update this morning and is now exhibiting the same issue. I'll do a capture as suggested.

gregc

Issue ended up being with FOP1 interfering. I have it disabled now and things look to be working normally.

Thank you for the online chat help!!

fme

Same problem here.
The panel hangs until a manual browser refresh with some browsers.

Tested with Chrome 11.0.696.57 beta and Firefox 4 on openSuse 11.4, IE8 and Firefox 3.6 on Windows 7 and IE7 on Vista.

Works with:

Firefox 4 (openSuse 11.4)
IE8 and Firefox 3.6 (Windows 7)

Works NOT with:

Chrome 11.0.696.57 beta (openSuse 11.4)
IE7 (Vista)

nicolas

Thanks for taking the time to make all those tests. But unfortunately none of them will help much if you do not at least open the javascript console on an affected browser and send me the output. Chrome has such a console.

Best regards,

swiltse

I'm experiencing similar issues with Chrome 11.0.696.65 not responding after a while on a WinXP machine. Here's the info from the console:

wav false
set session context
formato wav
set menu
Client has HTML5 web sockets!
setLang
set menu
set menu queue
no tiene permiso chat
get cookie extensionlist
get cookie queuelist
get cookie trunklist
get cookie conferencelist
get cookie parklist
Client has HTML5 web sockets!
set session extension
/fop2/fop2-variablesGENERAL.txtFailed to load resource: the server responded with a status of 404 (Not Found)
Second Init after loading variables
Attempt to connect to port 4445
12
intento conectar web socket en ws://sip.paviliongift.com:4445
on open
go
ws send <msg data="GENERAL|contexto|1|" />
ws connect 1
0,key=reTbUlZklQicDOALAMquHC7neiU en slot
0,version=2.20!!1 en slot
ws send <msg data="1|auth|149|79c9ab4cb9b0eee1b7ddaacb20c8398f" />
0,preferences=eyAiZGlzcGxheVF1ZXVlIjogIm1heCIsImR5bmFtaWNMaW5lRGlzcGxheSI6ICJvZmYiLCJsYW5ndWFnZSI6ICJlbiIsIm5vdGlmeUR1cmF0aW9uIjogIjYiLCJzb3VuZENoYXQiOiAiIiwic291bmRRdWV1ZSI6ICIiLCJzb3VuZFJpbmciOiAiIn0= en slot
llegaron las preferences { "displayQueue": "max","dynamicLineDisplay": "off","language": "en","notifyDuration": "6","soundChat": "","soundQueue": "","soundRing": ""}
Ponemos preferencias en valores default
0,vmailpath=/var/spool/asterisk/voicemail en slot
0,permit=YWxs en slot
set session phonebook
set session admin
2drawButton
set menu queue
no tiene licencia chat
ws send <msg data="1|initState||79c9ab4cb9b0eee1b7ddaacb20c8398f" />
0,demo=2 en slot 0
2,qualify=notok en slot 0
3,voicemail=2 en slot
3,voicemailcount=&vmail_count!0!1 en slot
3,qualify=ok en slot 0
4,status=unpaused en slot 0
4,rename=125 Ann Koehler en slot
4,qualify=ok en slot 0
5,qualify=ok en slot 0
6,qualify=ok en slot 0
8,status=unpaused en slot 0
8,rename=143 Daniel VerWeire en slot
8,voicemail=1 en slot
8,voicemailcount=&vmail_count!2!0 en slot
8,qualify=ok en slot 0
9,status=unpaused en slot 0
9,qualify=ok en slot 0
10,status=unpaused en slot 0
10,rename=152 Jerah Augello en slot
10,qualify=ok en slot 0
11,status=unpaused en slot 0
11,rename=153 Norma Pagano en slot
11,qualify=ok en slot 0
12,details= en slot 0
13,queuemembers=PGRpdiBjbGFzcz0nbWVtYmVycmVhZHkgbXljbGljaycgaWQ9J3FtIVFVRVVFLzI1MCFMb2NhbC8xNDlAZnJvbS1xdWV1ZS9uJz48L2Rpdj48c3Bhbj5TaGF3biBXaWx0c2U8L3NwYW4+PGJyIGNsYXNzPSdjbGVhcicvPg== en slot
14,queuemembers=PGRpdiBjbGFzcz0nbWVtYmVycmVhZHkgbXljbGljaycgaWQ9J3FtIVFVRVVFLzI1MSFMb2NhbC8xMjVAZnJvbS1xdWV1ZS9uJz48L2Rpdj48c3Bhbj5Bbm4gS29laGxlcjwvc3Bhbj48YnIgY2xhc3M9J2NsZWFyJy8+PGRpdiBjbGFzcz0nbWVtYmVycmVhZHkgbXljbGljaycgaWQ9J3FtIVFVRVVFLzI1MSFMb2NhbC8xNTJAZnJvbS1xdWV1ZS9uJz48L2Rpdj48c3Bhbj5KZXJhaCBBdWdlbGxvPC9zcGFuPjxiciBjbGFzcz0nY2xlYXInLz48ZGl2IGNsYXNzPSdtZW1iZXJyZWFkeSBteWNsaWNrJyBpZD0ncW0hUVVFVUUvMjUxIUxvY2FsLzE1M0Bmcm9tLXF1ZXVlL24nPjwvZGl2PjxzcGFuPk5vcm1hIFBhZ2Fubzwvc3Bhbj48YnIgY2xhc3M9J2NsZWFyJy8+ en slot
15,queuemembers= en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
4,settimer=0@UP en slot 1
4,state=UP en slot 1
4,settext=125 device en slot 1
4,direction=outbound en slot 1
4,settext=915073764556 en slot 1
4,settext=ZultysMX25/15073764556 en slot 1
4,direction=outbound en slot 1
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
4,settimer=0@UP en slot 1
4,state=UP en slot 1
4,settext=915073764556 en slot 1
4,settext=915073764556 en slot 1
4,link=915073764556 en slot 1
4,settimer=0@STOP en slot 1
4,settext=&inactive_line!1 en slot 1
4,state=DOWN en slot 1
DOWN pero quedan sesiones, no pongo boton en free
4,state=DOWN en slot 0
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot
4,qualify=ok en slot 0
ws send <msg data="1|ping||" />
envio ping 2
0,pong=0 en slot

nicolas

For everyone experience this issue, I think I found the cause. I will post a new beta tomorrow that you will be able to try, the problem is that the beta has a ton of new major features , so expect a ton of new bugs.

If you want to fix it you must restart fop2 server, the problem seems to occur when asterisk is restarted while there are fop2 clients connected.

Brief explanation of what it might be happening:

Upon fop2 start, he order of the socket connections is this:

Connection to AMI (server)
[More AMI Connections on multi server]
Connection to Client 1
Connection to Client 2
etc...

When the connection to AMI is lost (asterisk restarted or crashed), and if you have clients connected to fop2, the order changes and you end up with this:

Connection to Client 1
Connection to Client 2
Connetion to AMI (server)

Now, if a new client connects, you will have 2 clients , 1 manager and a final client connection. In those cases , not sure if the first or the last connections after AMI will stop receiving updates.

Connection to Client 1
Connection to Client 2
Connetion to AMI (server)
Connection to Client 3

Restarting fop2 server will clear the issue as the socket order is restored. The next beta includes a fix that will make all clients receive updates (it will not clear the event cache after each AMI read cycle)

compassist

Hi, I'm having the same issue with fop2.20 (basic license). Could you link/email me a copy of the beta version to test out? Thanks,

~ca

nicolas

http://www.fop2.com/downloads/fop2-2.21 ... 5-i386.tgz

compassist

Thanks, that seems to have fixed the problem with fop2 stopping.

However, there are two issues that we've noticed. One is that sometimes parked callers aren't removed from fop (we have someone who's supposedly been parked for an hour and a half, but picking up the line gives "no call parked). Sometimes it works fine thought.

The other is that only our IAX trunk shows calls, the ZAP trunks we have constly remain green, and never show a call, despite sometimes being in use.

Thanks,

~ca

FOP2 Community Forum

FOP2 Stops responding after awhile