Unable to establish WebSocket connection through Google Cloud Application Load Balancer

I am trying to configure WebSocket connections for my application hosted on Google Cloud using a Global External Application Load Balancer, but I am facing issues. Here's my setup:

Setup:

1. I have a single VM instance running two services:

   • Authentication Service on port 8080, accessible via https://auth.forge-code.com.
   • Notification Service on port 8081, accessible via https://notify.forge-code.com.

2. The notification service exposes a WebSocket endpoint on /ws/* (e.g., wss://notify.forge-code.com/ws/546).

3. I am using a Global External Application Load Balancer with the following configuration:

   • Frontend:
     • Protocol: HTTPS
     • HTTP keepalive timeout: 1200 seconds
     • SSL certificates configured for both subdomains (auth.forge-code.com, notify.forge-code.com).
   • Routing Rules:
     • auth.forge-code.com routes all traffic to auth-backend.
     • notify.forge-code.com/ws/* routes traffic to notify-backend.
     • Other traffic to notify.forge-code.com routes to notify-backend.
   • Backends:
     • auth-backend:
       • Port: 8080
       • Timeout: 30 seconds
     • notify-backend:
       • Port: 8081
       • Timeout: 30 seconds
       • Cloud CDN: Enabled
       • Logging: Enabled (sample rate: 1).

4. DNS resolves auth.forge-code.com and notify.forge-code.com correctly to the load balancer's IP.

The Problem

• HTTP requests to endpoints like https://notify.forge-code.com/health work perfectly.
• WebSocket connections to wss://notify.forge-code.com/ws/546 fail with the following errors:
  • Postman: 502 Bad Gateway
  • Browser console (sometimes): socket hang up.
• Direct WebSocket connections to the backend (ws://34.133.145.146:8081/ws/546) succeed.

Additional Information:

• Backend service is implemented in Rust using ntex and logs indicate that WebSocket connections are dropped as soon as they are initiated via the load balancer.
• I suspect the issue occurs when traffic returns from the backend to the load balancer, but I’m not sure how to debug this further.

Logs:

Here’s a snippet of the backend logs when the connection is dropped:

[INFO  ntex::web::middleware::logger] 35.191.206.16:41366 "GET /health HTTP/1.1" 200 47 "-" "GoogleHC/1.0" 0.000069
[INFO  notify_service::services::websocket_service] Client disconnected: 546
[INFO  notify_service::services::websocket_service] Connection dropped, stopping heartbeat task.

1. Postman connecting to wss://notify.forge-code.com/ws/546:

• Error: 502 Bad Gateway
• Handshake Details:
  • Request:

    GET /ws/546 HTTP/1.1
    Host: notify.forge-code.com
    Connection: Upgrade
    Upgrade: websocket
    Sec-WebSocket-Version: 13
    Sec-WebSocket-Key: x14nFpQQ/RYw17KRaAYteA==
    Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
    

  • Response:

    HTTP/1.1 502 Bad Gateway
    Content-Length: 110
    Content-Type: text/plain
    Connection: close
    Via: 1.1 google
    Date: Wed, 18 Dec 2024 10:50:58 GMT
    Alt-Svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000
    

2. Direct Connection to Backend (ws://34.133.145.146:8081/ws/546):

• Handshake Details:
  • Request:

    GET /ws/546 HTTP/1.1
    Host: 34.133.145.146:8081
    Connection: Upgrade
    Upgrade: websocket
    Sec-WebSocket-Version: 13
    Sec-WebSocket-Key: HBB28rJBqzubIZpcjX4IFQ==
    Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
    

  • Response:

    HTTP/1.1 101 Switching Protocols
    Connection: Upgrade
    Upgrade: websocket
    Sec-WebSocket-Accept: BMDxeu6H/GzSXaTDwL4/gunqy1A=
    Date: Wed, 18 Dec 2024 10:51:19 GMT
    

3. Postman connecting to ws://notify.forge-code.com/ws/546:

• Error: socket hang up
• Handshake Details:
  • Request:

    GET /ws/546 HTTP/1.1
    Host: notify.forge-code.com
    Connection: Upgrade
    Upgrade: websocket
    Sec-WebSocket-Version: 13
    Sec-WebSocket-Key: KgravwbL2GoTBZkgHj1uFQ==
    Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
    

  • No Response.

Notes:

• Edited to clarify that there is no 400 error.
• Included logs and handshake details as requested.

Questions

1. Why are WebSocket connections failing through the load balancer, even though HTTP traffic and direct WebSocket connections to the backend work fine?
2. How can I resolve this issue so that WebSocket connections through the load balancer succeed?