Well I've continued to work on this problem and managed to find a reasonable workaround/hack: turn off SSL for the websocket service and stick it behind an nginx reverse proxy to handle that part.
I removed reference to the certificate/key in fop2.cfg, and in my nginx config I added something along these lines:
server {
listen 4444 ssl;
ssl_certificate fullchain.pem;
ssl_certificate_key privkey.pem;
location / {
proxy_pass http://localhost:4445;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
Then added an iptables rule like so:
iptables -t nat -A PREROUTING -i <inbound interface> -p tcp --dport 4445 -j DNAT --to-destination :4444
The fop2 websocket service still listens on port 4445, but inbound connections for that port are caught by the iptables rule and forwarded to port 4444 instead where nginx handles them. nginx handles the SSL part and then proxys the connection to port 4445 on localhost, and since this traverses the loopback interface the iptables rule doesn't affect it so it hits the fop2 service as normal. This part might not be necessary, I did it because I assumed changing the bind port in fop2.cfg would probably also affect the port it pushes to your browser to make the websocket connection. Maybe someone else could iterate and improve on this.
I'd still be interested to know if there's a proper way to get this working without having to resort to such hacks though.