Cases2 cases

1 to 2 of 2

1. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query

   OpenAI2024

   Symptoms

   On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

   High error rateDatabase overload

   Trigger

   A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerApplication database

   Contributing factor

   No staged rolloutAutomation amplification

   Action items

   • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
   • Change service software to start faster and remove the first-start query, relieving the persistence layer
   • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
   • Remove the expensive database query from critical startup paths
   • Add monitoring and alerting for networking control plane issues
   • Introduce staged rollouts with longer soak time for infrastructure changes
   • Audit systems for other slow queries that may affect service start time

   Lessons

   A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

   View case
2. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx

   LINE+2022

   Symptoms

   The graph of WebSocket connections to messaging-hub, the messaging platform used by the Japanese food delivery service Demaecan, showed sudden deep drops even outside peak traffic. The drops kept coming back at irregular intervals. Messages still reached clients, because messaging-hub switches to Firebase Cloud Messaging when a WebSocket is gone.

   Connection resetsDatabase overloadIntermittent connection failuresHigh latency

   Trigger

   Another service that shared the same L7 load balancer deployed its own load balancer configuration in the Verda dashboard, which reloaded HAProxy; 30 minutes later, at the hard-stop-after limit, the old HAProxy process exited and closed every WebSocket connection it still held.

   Root cause

   Configuration change

   Component

   HAProxyCloud load balancerconntrack tableIngress controller

   Contributing factor

   Change with a delayed effectExcessive blast radiusRetry stormUnverified assumption

   Detection

   The developer noticed deep, sudden drops in the WebSocket connection graph that were not tied to peak traffic and kept recurring irregularly. The Nginx packet drops were found through network monitoring.

   Remediation

   • separated the Merchant system's token issuance API into its own deployment so reconnect traffic did not delay other Merchant APIs
   • scaled out Nginx with pod anti-affinity and a label selector so each pod ran on a different node, which stopped the nf_conntrack packet drops
   • raised Ingress Nginx proxy-read-timeout and proxy-send-timeout and the load balancer timeout to 3600s, which did not stop the drops
   • moved messaging-hub from the shared L7 load balancer to an L4 load balancer, which ended the WebSocket drops

   Lessons

   A reload of a shared load balancer by a different team can cut long-lived connections, and with hard-stop-after the cut comes 30 minutes later, far from the change that caused it. The author's lesson is not to narrow the scope of the search too early: trusting that the infrastructure was fine kept the investigation inside the team's own systems for days. A reconnect wave from many WebSocket clients can overload the systems that issue connection tokens and fill nf_conntrack on proxy nodes. Scaling Nginx out only helps for conntrack if the pods land on different nodes, and anti-affinity leaves extra replicas Pending once they outnumber the nodes. That messaging-hub problems can still load the connected systems remains an open problem.

   View case

By root cause2