Cases4 cases

1 to 4 of 4

1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services

   Grafana Labs2025

   Symptoms

   At 08:41 UTC on 18 February 2025, several Grafana Labs application teams noticed that some of their endpoints were no longer available. The investigation quickly found that the load balancers for those endpoints were gone.

   Service outageResources deleted

   Trigger

   A change to the merge policy of a Crossplane composition, made so that the default TLS policy of the load balancers could be overridden, was applied to development, staging and production at the same time.

   Root cause

   Configuration change

   Component

   Configuration templatingCloud load balancer

   Contributing factor

   Unverified assumptionNo staged rolloutExcessive blast radiusNo safeguard on destructive change

   Detection

   Application teams noticed at 08:41 UTC that some of their endpoints were no longer available.

   Remediation

   • rolled back the Crossplane composition change
   • recreated the affected Kubernetes Services so that Kubernetes recreated their load balancers

   Action items

   • Add a CI check that blocks a deployment to more than one wave, so any exception to the dev, staging, canary, production order is caught
   • Give higher priority to a more sophisticated wave design that was already under discussion
   • Enable deletion protection on the load balancers, one layer below Crossplane
   • Enable deletion protection on other critical cloud provider resources
   • Develop the Crossplane Managed Resources exporter for change validation and visibility, now in beta testing within Platform engineering; simulated against the change that caused this outage, it shows a clear diff
   • Complete and test support for reconfiguring the TLS policy so application teams can deploy it

   Lessons

   A change that looked small and low-risk had a side effect that only a full end-to-end test through Crossplane and the cloud provider's load balancer controller would have shown; seeing the intended override work was not that test. Staged rollout and blast-radius limits that exist only as process can be skipped, so Grafana Labs moved to enforce waves in CI and to add deletion protection on the load balancers themselves, beneath the Crossplane layer.

   View case
2. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx

   LINE+2022

   Symptoms

   The graph of WebSocket connections to messaging-hub, the messaging platform used by the Japanese food delivery service Demaecan, showed sudden deep drops even outside peak traffic. The drops kept coming back at irregular intervals. Messages still reached clients, because messaging-hub switches to Firebase Cloud Messaging when a WebSocket is gone.

   Connection resetsDatabase overloadIntermittent connection failuresHigh latency

   Trigger

   Another service that shared the same L7 load balancer deployed its own load balancer configuration in the Verda dashboard, which reloaded HAProxy; 30 minutes later, at the hard-stop-after limit, the old HAProxy process exited and closed every WebSocket connection it still held.

   Root cause

   Configuration change

   Component

   HAProxyCloud load balancerconntrack tableIngress controller

   Contributing factor

   Change with a delayed effectExcessive blast radiusRetry stormUnverified assumption

   Detection

   The developer noticed deep, sudden drops in the WebSocket connection graph that were not tied to peak traffic and kept recurring irregularly. The Nginx packet drops were found through network monitoring.

   Remediation

   • separated the Merchant system's token issuance API into its own deployment so reconnect traffic did not delay other Merchant APIs
   • scaled out Nginx with pod anti-affinity and a label selector so each pod ran on a different node, which stopped the nf_conntrack packet drops
   • raised Ingress Nginx proxy-read-timeout and proxy-send-timeout and the load balancer timeout to 3600s, which did not stop the drops
   • moved messaging-hub from the shared L7 load balancer to an L4 load balancer, which ended the WebSocket drops

   Lessons

   A reload of a shared load balancer by a different team can cut long-lived connections, and with hard-stop-after the cut comes 30 minutes later, far from the change that caused it. The author's lesson is not to narrow the scope of the search too early: trusting that the infrastructure was fine kept the investigation inside the team's own systems for days. A reconnect wave from many WebSocket clients can overload the systems that issue connection tokens and fill nf_conntrack on proxy nodes. Scaling Nginx out only helps for conntrack if the pods land on different nodes, and anti-affinity leaves extra replicas Pending once they outnumber the nodes. That messaging-hub problems can still load the connected systems remains an open problem.

   View case
3. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404

   DFDS2021

   Symptoms

   At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.

   Service outageHigh error rateProbe failures

   Trigger

   The automated Kubernetes node rollover on 2021-09-06 restarted all the Traefik v2 pods, five days after a malformed IngressRoute manifest had been applied.

   Root cause

   Software defect

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Change with a delayed effectNo configuration testsExcessive blast radius

   Detection

   All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.

   Remediation

   • changed the ALB health check path, which made the targets healthy but did not stop the 404s
   • rolled back by hand to Traefik v1 as the main ingress controller
   • deleted the malformed IngressRoute by hand and confirmed the Traefik v2 bug
   • applied the changes upstream

   Action items

   • Upgrade Traefik to v2.5, where the bug is fixed, and verify that it removes the bug
   • Change the Traefik v2 health check path to something more robust
   • Set up an external liveness probe on the ingress controller
   • Consider workload segmentation to limit the blast zone
   • Look into monitoring the Traefik v2 API for rapid changes
   • Look into smoke testing during and after node rollover

   Lessons

   A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik. On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.

   View case
4. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard

   JW Player2019

   Symptoms

   Datadog alerted the DevOps team to a high normalised load average, first on the staging cluster (Day 0, 21:06 EST) and the next day on the development cluster (Day 1, 16:53 EST). The load was first put down to a legitimate service that kept scaling up and down in both environments, and because of the volume of identical alerts they were muted until things settled.

   Sustained high node CPUUnauthorised access or a foreign workload

   Trigger

   An automated crawler found the Weave Scope dashboard on a public load balancer that required no authentication. Weave Scope can open a shell in running containers, and its scope-agent containers ran privileged, as root and with the host's root filesystem mounted, so a command started there could run on the host.

   Root cause

   Configuration change

   Component

   Cluster dashboardCloud load balancer

   Contributing factor

   Ignored early signalUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   Datadog alerts on high normalised load average, muted at first as noise from a legitimate service; on Day 3 an engineer ran top on a node and found the gcc process at 100% CPU.

   Remediation

   • stopped Weave Scope and removed its deployment from all Kubernetes clusters
   • checked access logs for unauthorised access to the instances
   • confirmed the miner's outbound connections went only to a mining pool
   • isolated one affected node for later investigation
   • rotated out every node in each cluster so all affected hosts were destroyed and rebuilt

   Action items

   • Audit all Kubernetes and deployment YAML files and security groups for services that are unintentionally public or misconfigured
   • Replace load-average alerts with more meaningful monitors and tune them to signal a real anomaly
   • Research tools such as falcon or sysdig for behavioural monitoring and anomaly and intrusion detection
   • Evaluate Istio and Linkerd for observing and controlling end-to-end network traffic
   • Analyse the cost of scanning Docker images and containers for known vulnerabilities
   • Set aside time in architecture discussions to revisit older choices such as running containers as root
   • Share knowledge of large projects, such as Kubernetes version migrations, across the whole team

   Lessons

   Load average was not a useful signal for spotting a problem in the clusters, and muted noisy alerts nearly hid the miner. A load balancer that Kubernetes creates belongs to Kubernetes: a manual security group edit in the AWS console is reverted, so internal exposure must be declared in the Kubernetes YAML. A dashboard that can open shells, in front of privileged agents that mount the host's root filesystem, gives whoever reaches it root on every node.

   View case

By root cause2