Cases3 cases

1 to 3 of 3

1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services

   Grafana Labs2025

   Symptoms

   At 08:41 UTC on 18 February 2025, several Grafana Labs application teams noticed that some of their endpoints were no longer available. The investigation quickly found that the load balancers for those endpoints were gone.

   Service outageResources deleted

   Trigger

   A change to the merge policy of a Crossplane composition, made so that the default TLS policy of the load balancers could be overridden, was applied to development, staging and production at the same time.

   Root cause

   Configuration change

   Component

   Configuration templatingCloud load balancer

   Contributing factor

   Unverified assumptionNo staged rolloutExcessive blast radiusNo safeguard on destructive change

   Detection

   Application teams noticed at 08:41 UTC that some of their endpoints were no longer available.

   Remediation

   • rolled back the Crossplane composition change
   • recreated the affected Kubernetes Services so that Kubernetes recreated their load balancers

   Action items

   • Add a CI check that blocks a deployment to more than one wave, so any exception to the dev, staging, canary, production order is caught
   • Give higher priority to a more sophisticated wave design that was already under discussion
   • Enable deletion protection on the load balancers, one layer below Crossplane
   • Enable deletion protection on other critical cloud provider resources
   • Develop the Crossplane Managed Resources exporter for change validation and visibility, now in beta testing within Platform engineering; simulated against the change that caused this outage, it shows a clear diff
   • Complete and test support for reconfiguring the TLS policy so application teams can deploy it

   Lessons

   A change that looked small and low-risk had a side effect that only a full end-to-end test through Crossplane and the cloud provider's load balancer controller would have shown; seeing the intended override work was not that test. Staged rollout and blast-radius limits that exist only as process can be skipped, so Grafana Labs moved to enforce waves in CI and to add deletion protection on the load balancers themselves, beneath the Crossplane layer.

   View case
2. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx

   LINE+2022

   Symptoms

   The graph of WebSocket connections to messaging-hub, the messaging platform used by the Japanese food delivery service Demaecan, showed sudden deep drops even outside peak traffic. The drops kept coming back at irregular intervals. Messages still reached clients, because messaging-hub switches to Firebase Cloud Messaging when a WebSocket is gone.

   Connection resetsDatabase overloadIntermittent connection failuresHigh latency

   Trigger

   Another service that shared the same L7 load balancer deployed its own load balancer configuration in the Verda dashboard, which reloaded HAProxy; 30 minutes later, at the hard-stop-after limit, the old HAProxy process exited and closed every WebSocket connection it still held.

   Root cause

   Configuration change

   Component

   HAProxyCloud load balancerconntrack tableIngress controller

   Contributing factor

   Change with a delayed effectExcessive blast radiusRetry stormUnverified assumption

   Detection

   The developer noticed deep, sudden drops in the WebSocket connection graph that were not tied to peak traffic and kept recurring irregularly. The Nginx packet drops were found through network monitoring.

   Remediation

   • separated the Merchant system's token issuance API into its own deployment so reconnect traffic did not delay other Merchant APIs
   • scaled out Nginx with pod anti-affinity and a label selector so each pod ran on a different node, which stopped the nf_conntrack packet drops
   • raised Ingress Nginx proxy-read-timeout and proxy-send-timeout and the load balancer timeout to 3600s, which did not stop the drops
   • moved messaging-hub from the shared L7 load balancer to an L4 load balancer, which ended the WebSocket drops

   Lessons

   A reload of a shared load balancer by a different team can cut long-lived connections, and with hard-stop-after the cut comes 30 minutes later, far from the change that caused it. The author's lesson is not to narrow the scope of the search too early: trusting that the infrastructure was fine kept the investigation inside the team's own systems for days. A reconnect wave from many WebSocket clients can overload the systems that issue connection tokens and fill nf_conntrack on proxy nodes. Scaling Nginx out only helps for conntrack if the pods land on different nodes, and anti-affinity leaves extra replicas Pending once they outnumber the nodes. That messaging-hub problems can still load the connected systems remains an open problem.

   View case
3. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard

   JW Player2019

   Symptoms

   Datadog alerted the DevOps team to a high normalised load average, first on the staging cluster (Day 0, 21:06 EST) and the next day on the development cluster (Day 1, 16:53 EST). The load was first put down to a legitimate service that kept scaling up and down in both environments, and because of the volume of identical alerts they were muted until things settled.

   Sustained high node CPUUnauthorised access or a foreign workload

   Trigger

   An automated crawler found the Weave Scope dashboard on a public load balancer that required no authentication. Weave Scope can open a shell in running containers, and its scope-agent containers ran privileged, as root and with the host's root filesystem mounted, so a command started there could run on the host.

   Root cause

   Configuration change

   Component

   Cluster dashboardCloud load balancer

   Contributing factor

   Ignored early signalUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   Datadog alerts on high normalised load average, muted at first as noise from a legitimate service; on Day 3 an engineer ran top on a node and found the gcc process at 100% CPU.

   Remediation

   • stopped Weave Scope and removed its deployment from all Kubernetes clusters
   • checked access logs for unauthorised access to the instances
   • confirmed the miner's outbound connections went only to a mining pool
   • isolated one affected node for later investigation
   • rotated out every node in each cluster so all affected hosts were destroyed and rebuilt

   Action items

   • Audit all Kubernetes and deployment YAML files and security groups for services that are unintentionally public or misconfigured
   • Replace load-average alerts with more meaningful monitors and tune them to signal a real anomaly
   • Research tools such as falcon or sysdig for behavioural monitoring and anomaly and intrusion detection
   • Evaluate Istio and Linkerd for observing and controlling end-to-end network traffic
   • Analyse the cost of scanning Docker images and containers for known vulnerabilities
   • Set aside time in architecture discussions to revisit older choices such as running containers as root
   • Share knowledge of large projects, such as Kubernetes version migrations, across the whole team

   Lessons

   Load average was not a useful signal for spotting a problem in the clusters, and muted noisy alerts nearly hid the miner. A load balancer that Kubernetes creates belongs to Kubernetes: a manual security group edit in the AWS console is reverted, so internal exposure must be declared in the Kubernetes YAML. A dashboard that can open shells, in front of privileged agents that mount the host's root filesystem, gives whoever reaches it root on every node.

   View case

By root cause1