Slow and unstable connection to panel

  1. 4 years ago
    Edited 4 years ago by nikriaz

    Hello, I'm struggling with slow connection to the Panel. When I try to connect I get "Connecting to server, attempt number : ...1" and good if I will connect over 20-30 counts. Eventually, connect will happen but become unstable. Clicking on any button may cause re-connection again (but may be not).

    What is even more strange, Panel has tend to be "warming up". If I more often manually re-connect to panel and more frequent clicking on buttons, panel becomes more responsive and stable...

    I run Elastix 2.4 and recently upgraded to FOP2 2.29 latest but this problem has been persistent all way down to Elastix 2.4 and FOP2 2.0. At the very initial stage it worked well but then something happened.
    FOP1 worked just fine.

    I carefully reviewed all related posts:

    • port 4445 is open and listening
    • FOP1 is disabled
    • Firewall (iptables) switched off
    • excessive events in manager.conf fired off
    • workstation is connected to the server over direct Ethernet cable (only switch between, no active devices)
    • same problem in Chrome and Firefox
    • I use HTTPS, probably because it's default for Elastix I don't know how to bypass HTTPS and connect via HTTP, it simply doesn't work.
    • Chrome extension seems work smooth

    Any ideas?

    Writing the solution for posterity an other users that could be affected. Looking at the FOP2 debug log I found this error:

    callback (ignoring): /etc/pki/tls/certs/localhost.crt: failed to use local certificate chain (cert_file or cert)

    FOP2 knows how to handle certificate files in PEM format (the most common, that is base64 encoded).

    nikriaz had a custom certificate in another (binary) format (DEM). FOP2 was not able to handle that certificate so it did not negotiate SSL. Converting it to PEM format made it work.

    The command to convert a certificate from DEM to PEM:

    openssl x509 -inform der -in /etc/pki/tls/certs/localhost.crt -out /etc/pki/tls/certs/localhost.pem

    Then changing the file to read on fop2.cfg

    ssl_certificate_file=/etc/pki/tls/certs/localhost.pem

    (As I did not want to alter the original certificate that might be used by other software and perhaps only in that format).

    With the file in the correct format FOP2 knows how to handle it.

  2. admin

    20 May 2015 Administrator
    Edited 4 years ago by admin

    I have an idea about the slow initial connectivity, but not what happens later on. Websocket protocol has evolved, implementations in browsers are following RFC a little more closely with each release. One of those changes is the enforcement of secure web sockets (wss) when the HTTP connection is also secure/sslized. That means that if you connect to an application using https, and the application attempts a web socket connection, it *must* be secured with SSL. A standard ws connection over HTTPS will fail. This did not happen with websocket initial implementation in browsers and was gradually being adapted first in Firefox, then in Chrome.

    FOP2 knows every Websocket protocol, and it also uses flash xmlsockets as a fallback mechanism (and it was initially the only protocol before Websocket even existed). So, if you do not have ssl certificates configured for the fop2 server, it will offer only normal websocket , the browser will fail to negotiate secure web sockets, then it will attempt normal websocket and browser will make it fail, and finally it will attempt flash xmlsockets and it will work (but it takes several attemps/fallbacks until the connection is finally done).

    What this means is that in order to have a fast negotiation of a web socket connection with FOP2 when using https, you *must* configure the ssl certificates in fop2.cfg. Secure web socket was added in FOP 2.27, but usually when you upgrade, your original fop2.cfg file is preserved, and the new one (with new options) is saved as fop2.cfg.new. (So you must edit fop2.cfg by hand and add the proper options).

    If you come from Elastix, then you might have installed the Elastix version of FOP2 also, that version has configuration files in a different place than the standard FOP2 install.. so, check the contents of the file /etc/sysconfig/fop2. If in that file you have -c /etc/asterisk/fop2 as one of the options, the the config file is in /etc/asterisk/fop2 otherwise it is in /usr/local/fop2. Open the file (fop2.cfg) and add the ssl certtificate configurations (they must match exactly the certificates that are set in your web browser), in an Elastix install the default values are:

    ssl_certificate_file=/etc/pki/tls/certs/localhost.crt ssl_certificate_key_file=/etc/pki/tls/private/localhost.key

    With that in place, after restarting FOP2, the initial connection to it over https should be fast and not take more than one cycle/attempt.

    Now, this does not affect at all the 'stability' you mention. That is something entirely different and I do not know what can cause that, except for some kind of network issue between the browser and the server. FOP2 uses different keepalive mechanisms besides the actual protocol keepalives, and it will reconnect if those fail.

  3. Hello, thank you for the great answer.
    Well, I re-intsalled FOP2 step-by-step from your site so not sure if something left from Elastix.

    In /etc/sysconfig/fop2 I have only OPTIONS="-d"
    In /etc/asterisk/fop2 I have nothing
    In /usr/local/fop2 I have normal fop2.cfg; both strings are in place.
    I double checked; both certificates are in place.

    We have Certificate Server (from Microsoft) in our environment so I issued and installed certificates.
    It seems they work in Elastix. The only issue that my certificate chain doesn't have intermediate server and I don't have published CRL. Both Chrome and Firefox consider this as unsecure but skip usual additional clicks to open "unsecure" sites.

    Any chance to check if my certificates are okey for FOP2? In some Microsoft products. lack of CRL is an issue.
    Could you advice, why I can't connect over HTTP at all?

  4. In Chrome I have also unusual "shield" sign on the right hand side from address bar.
    It says "this page attempts to load unsecured scripts" If I click "load unsecured scripts" it triggers long reconncetion cycle. Could it say something?

  5. admin

    20 May 2015 Administrator

    Problem is in the browser implementation of secure web sockets, they do not display a warning when an insecure certificate is being used, and you still need to allow that somehow. You can try to open the connection like this:

    https://your.server:4445

    That should prompt you for the insecure certificate, that you must allow. After that, the background websocket connection when accessing fop2 normally should work.

    in any case, you can open the javascript console in chrome while attempting connection, and you will see every connection attempt, the method, and fallbacks..

  6. Edited 4 years ago by admin

    Wow, it was really cool advice. Just in case, Java Console could be opened in Chrome via Shift+Ctrl+J
    Could you look into???

    So, I have following errors:

    Failed to load resource: the server responded with a status of 404 (Not Found)
    fail fop2-variables, default to port 4445 wit no TLS
    antes de preinit fail fop2variables
    pre init
    Client has HTML5 web sockets!
    connectxml
    intento conectar web socket en wss://my.server:4445
    set session context

    ---- Now following connection attempts, counts of 11---

    WebSocket connection to 'wss://my.server:4445/' failed: Error in connection establishment: net::ERR_TIMED_OUT
    WebSocket Error
    could not connect via wss, attempt ws

    ---- Now following connection attempts, counts of 5 ---

    intento conectar web socket en ws://my.server:4445
    Mixed Content: The page at 'https://my.server/fop2/?exten=1200&pass=xxxx' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://my.server:4445/'. This request has been blocked; this endpoint must be available over WSS.
    WebSocket Error
    could not connect via ws, attempt flash xmlsockets

    ---- Now following connection attempts, counts of 10 ---

    connectxml
    embed flash
    Attempt flash xmlsocket connection on port 4445
    2
    Connection successful flash xmlsockets general

    -- Now there are a lot of stuff and ---

    GET https://my.server/fop2/fop2-variablesGENERAL.txt 404 (Not Found)
    fail fop2-variables, default to port 4445 wit no TLS
    antes de preinit fail fop2variables
    pre init
    Client has HTML5 web sockets!
    connectxml
    intento conectar web socket en wss://my.server:4445
    set session context

  7. admin

    20 May 2015 Administrator
    Edited 4 years ago by admin

    What you see is the typical failover when secure web sockets fails to negotiate.. fop2 server since version 2.27 supports it, but everything I said before still applies. My guess is that the ssued certificate was not accepted/allowed. You can contact me via the live help if you can give me ssh access to your box in order to test it out with you. I am online now.

    Best regards,

  8. Sorry, missed your kind answer due to time zone shift. Will try to reach you out today. I appreciate your help!

  9. admin

    21 May 2015 Administrator Answer
    Edited 4 years ago by admin

    Writing the solution for posterity an other users that could be affected. Looking at the FOP2 debug log I found this error:

    callback (ignoring): /etc/pki/tls/certs/localhost.crt: failed to use local certificate chain (cert_file or cert)

    FOP2 knows how to handle certificate files in PEM format (the most common, that is base64 encoded).

    nikriaz had a custom certificate in another (binary) format (DEM). FOP2 was not able to handle that certificate so it did not negotiate SSL. Converting it to PEM format made it work.

    The command to convert a certificate from DEM to PEM:

    openssl x509 -inform der -in /etc/pki/tls/certs/localhost.crt -out /etc/pki/tls/certs/localhost.pem

    Then changing the file to read on fop2.cfg

    ssl_certificate_file=/etc/pki/tls/certs/localhost.pem

    (As I did not want to alter the original certificate that might be used by other software and perhaps only in that format).

    With the file in the correct format FOP2 knows how to handle it.

  10. That was absolutely fabulous. Thank you!!!

or Sign Up to reply!