Slow and unstable connection to panel

Comments

• nicolas

  May 2015 edited May 2015

  I have an idea about the slow initial connectivity, but not what happens later on. Websocket protocol has evolved, implementations in browsers are following RFC a little more closely with each release. One of those changes is the enforcement of secure web sockets (wss) when the HTTP connection is also secure/sslized. That means that if you connect to an application using https, and the application attempts a web socket connection, it *must* be secured with SSL. A standard ws connection over HTTPS will fail. This did not happen with websocket initial implementation in browsers and was gradually being adapted first in Firefox, then in Chrome.
  
  FOP2 knows every Websocket protocol, and it also uses flash xmlsockets as a fallback mechanism (and it was initially the only protocol before Websocket even existed). So, if you do not have ssl certificates configured for the fop2 server, it will offer only normal websocket , the browser will fail to negotiate secure web sockets, then it will attempt normal websocket and browser will make it fail, and finally it will attempt flash xmlsockets and it will work (but it takes several attemps/fallbacks until the connection is finally done).
  
  What this means is that in order to have a fast negotiation of a web socket connection with FOP2 when using https, you *must* configure the ssl certificates in fop2.cfg. Secure web socket was added in FOP 2.27, but usually when you upgrade, your original fop2.cfg file is preserved, and the new one (with new options) is saved as fop2.cfg.new. (So you must edit fop2.cfg by hand and add the proper options).
  
  If you come from Elastix, then you might have installed the Elastix version of FOP2 also, that version has configuration files in a different place than the standard FOP2 install.. so, check the contents of the file /etc/sysconfig/fop2. If in that file you have -c /etc/asterisk/fop2 as one of the options, the the config file is in /etc/asterisk/fop2 otherwise it is in /usr/local/fop2. Open the file (fop2.cfg) and add the ssl certtificate configurations (they must match exactly the certificates that are set in your web browser), in an Elastix install the default values are:
  
  [fixed]
  ssl_certificate_file=/etc/pki/tls/certs/localhost.crt
  ssl_certificate_key_file=/etc/pki/tls/private/localhost.key
  [/fixed]
  
  With that in place, after restarting FOP2, the initial connection to it over https should be fast and not take more than one cycle/attempt.
  
  Now, this does not affect at all the 'stability' you mention. That is something entirely different and I do not know what can cause that, except for some kind of network issue between the browser and the server. FOP2 uses different keepalive mechanisms besides the actual protocol keepalives, and it will reconnect if those fail.
  
  
  
  
  
  
• nikriaz

  May 2015

  Hello, thank you for the great answer.
  Well, I re-intsalled FOP2 step-by-step from your site so not sure if something left from Elastix.
  
  In /etc/sysconfig/fop2 I have only OPTIONS="-d"
  In /etc/asterisk/fop2 I have nothing
  In /usr/local/fop2 I have normal fop2.cfg; both strings are in place.
  I double checked; both certificates are in place.
  
  We have Certificate Server (from Microsoft) in our environment so I issued and installed certificates.
  It seems they work in Elastix. The only issue that my certificate chain doesn't have intermediate server and I don't have published CRL. Both Chrome and Firefox consider this as unsecure but skip usual additional clicks to open "unsecure" sites.
  
  Any chance to check if my certificates are okey for FOP2? In some Microsoft products. lack of CRL is an issue.
  Could you advice, why I can't connect over HTTP at all?
• nikriaz

  May 2015

  In Chrome I have also unusual "shield" sign on the right hand side from address bar.
  It says "this page attempts to load unsecured scripts" If I click "load unsecured scripts" it triggers long reconncetion cycle. Could it say something?
• nicolas

  May 2015

  Problem is in the browser implementation of secure web sockets, they do not display a warning when an insecure certificate is being used, and you still need to allow that somehow. You can try to open the connection like this:
  
  https://your.server:4445
  
  That should prompt you for the insecure certificate, that you must allow. After that, the background websocket connection when accessing fop2 normally should work.
  
  in any case, you can open the javascript console in chrome while attempting connection, and you will see every connection attempt, the method, and fallbacks..
  
  
• nikriaz

  May 2015 edited May 2015

  Wow, it was really cool advice. Just in case, Java Console could be opened in Chrome via Shift+Ctrl+J
  Could you look into???
  
  So, I have following errors:
  
  Failed to load resource: the server responded with a status of 404 (Not Found)
  fail fop2-variables, default to port 4445 wit no TLS
  antes de preinit fail fop2variables
  pre init
  Client has HTML5 web sockets!
  connectxml
  intento conectar web socket en wss://my.server:4445
  set session context
  
  ---- Now following connection attempts, counts of 11---
  
  WebSocket connection to 'wss://my.server:4445/' failed: Error in connection establishment: net::ERR_TIMED_OUT
  WebSocket Error
  could not connect via wss, attempt ws
  
  ---- Now following connection attempts, counts of 5 ---
  
  intento conectar web socket en ws://my.server:4445
  Mixed Content: The page at 'https://my.server/fop2/?exten=1200&pass=xxxx' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://my.server:4445/'. This request has been blocked; this endpoint must be available over WSS.
  WebSocket Error
  could not connect via ws, attempt flash xmlsockets
  
  ---- Now following connection attempts, counts of 10 ---
  
  connectxml
  embed flash
  Attempt flash xmlsocket connection on port 4445
  2
  Connection successful flash xmlsockets general
  
  -- Now there are a lot of stuff and ---
  
  GET https://my.server/fop2/fop2-variablesGENERAL.txt 404 (Not Found)
  fail fop2-variables, default to port 4445 wit no TLS
  antes de preinit fail fop2variables
  pre init
  Client has HTML5 web sockets!
  connectxml
  intento conectar web socket en wss://my.server:4445
  set session context
• nicolas

  May 2015 edited May 2015

  What you see is the typical failover when secure web sockets fails to negotiate.. fop2 server since version 2.27 supports it, but everything I said before still applies. My guess is that the ssued certificate was not accepted/allowed. You can contact me via the live help if you can give me ssh access to your box in order to test it out with you. I am online now.
  
  Best regards,
• nikriaz

  May 2015

  Sorry, missed your kind answer due to time zone shift. Will try to reach you out today. I appreciate your help!
• nicolas

  May 2015 edited May 2015

  Writing the solution for posterity an other users that could be affected. Looking at the FOP2 debug log I found this error:
  
  [fixed]
  callback (ignoring): /etc/pki/tls/certs/localhost.crt: failed to use local certificate chain (cert_file or cert)
  [/fixed]
  
  FOP2 knows how to handle certificate files in PEM format (the most common, that is base64 encoded).
  
  nikriaz had a custom certificate in another (binary) format (DEM). FOP2 was not able to handle that certificate so it did not negotiate SSL. Converting it to PEM format made it work.
  
  The command to convert a certificate from DEM to PEM:
  

  openssl x509 -inform der -in /etc/pki/tls/certs/localhost.crt -out /etc/pki/tls/certs/localhost.pem
  

  
  Then changing the file to read on fop2.cfg
  
  [fixed]
  ssl_certificate_file=/etc/pki/tls/certs/localhost.pem
  [/fixed]
  
  (As I did not want to alter the original certificate that might be used by other software and perhaps only in that format).
  
  With the file in the correct format FOP2 knows how to handle it.
  
  
• nikriaz

  May 2015

  That was absolutely fabulous. Thank you!!!