Cases14 cases

9 to 13 of 14

1. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes

   DFDS2023sev1

   Symptoms

   At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.

   Service outageResources deletedMonitoring outage

   Trigger

   A scheduled node rollover had accidentally made containerd the default container runtime, which stopped log shipping to CloudWatch; the unscheduled rollover to revert it timed out, and the pipeline run that followed reduced the Auto Scaling group from 30 to 15 instances.

   Root cause

   Configuration change

   Component

   Container runtimeNode launch templateNode poolTerraformTelemetry agent

   Contributing factor

   Automation amplificationUnreviewed defaultsUnrepresentative stagingNo configuration testsNo runbookHarmful manual action

   Detection

   End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.

   Remediation

   • ran the node rollover script to add new nodes, which failed on a timeout from Docker Hub
   • ran the node rollover script a second time, which added new nodes that the load balancer marked healthy at 11:01 UTC

   Action items

   • Document the rollover process, especially what can be done when it goes wrong
   • Investigate options for improving the rollover process
   • Add fluentd log collection to the QA test suite so logs are covered end to end
   • Add alarms for fluentd when too little or no log data is sent
   • Implement a staging environment
   • Implement guards against selecting an invalid container runtime

   Lessons

   When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way. The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.

   View case
2. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down

   OpenAI2023

   Symptoms

   At 4:53 pm Pacific on 25 January 2023 one of OpenAI's GPU clusters had a major outage, and the text-davinci-003 model went fully down.

   Service outageThroughput degradation

   Trigger

   A configuration change to the CNI (Container Network Interface) plugins, which provide network connectivity to containers, was deployed to a GPU cluster that contained a hardware variant the staging environment did not have.

   Root cause

   Configuration change

   Component

   CNI plugin

   Contributing factor

   Unrepresentative stagingMonitoring blind spot

   Detection

   Engineers immediately identified the problem as network connectivity, but it took nearly an hour to identify CNI as the cause.

   Remediation

   • put a fix for the CNI configuration change in place, which restored 80% of traffic immediately
   • deployed models back onto GPUs over about three more hours until capacity was fully stable

   Action items

   • Make staging environments contain every type of hardware configuration so they better represent production
   • Monitor and alert on the health of the CNI system directly
   • Fix the network and storage bottlenecks that limit how quickly many replicas of a model can be deployed

   Lessons

   A change that had been deployed to other clusters over 24 hours and deemed safe still broke a cluster with different hardware, so staging needs every hardware configuration that production has.

   View case
3. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout

   Grafana Labs2022

   Symptoms

   During a rollout of the Grafana Loki distributor Deployment in one Grafana Cloud Logs environment, the write path went down for about 25 minutes. The newly started distributor pods were OOM-killed over and over, never passed their health and readiness checks, and the rollout stopped making progress.

   Service outageOOMKilledLoad imbalanceProbe failures

   Trigger

   Step 1 of moving the Loki ring off a separately deployed key-value store onto embedded memberlist added a new label, gossip_ring_member: 'true', to the distributor pods and Service, together with a small config change, and the change required a rollout of the distributors.

   Root cause

   Configuration change

   Component

   Configuration templatingService discoveryContainer memory limit

   Contributing factor

   Unreviewed defaultsIgnored early signalUnrepresentative staging

   Detection

   The first clue was kubectl get pods and the kube_deployment_status_replicas_updated metric showing that only 16 of 140 distributor pods had been updated.

   Remediation

   • scaled the distributor Deployment out horizontally so the load spread over more pods with the new label and the rollout could continue
   • reverted the migration step to end the outage
   • set service_ignored_labels:: [$._config.gossip_member_label] and passed it to k.util.serviceFor so the label is no longer copied into the Service selector

   Action items

   • Update the function that generates a Service from a Deployment or StatefulSet to handle edge cases, including this one

   Lessons

   A Service whose selector is built by copying every label of a workload changes which pods it matches as soon as a new label is added. During a rollout it then matches only the new pods, which receive all the traffic before there are enough of them. A few errors during a rollout in other environments hid the problem, because those environments gave distributors more memory and could absorb the shift. Manifest diffs kept for every pull request made it possible to see that the Service selector had changed.

   View case
4. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change

   UK Ministry of Justice2021

   Symptoms

   At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

   Deploys failing

   Trigger

   The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

   Root cause

   Breaking change in an upgrade

   Component

   Admission webhookIngress controller

   Contributing factor

   Unrepresentative staging

   Detection

   A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

   Remediation

   • rolled back the ModSec ingress controller module to v0.0.7

   Action items

   • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
   • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
   • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
   • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)

   View case
5. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order

   Skyscanner2019

   Symptoms

   On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.

   High latencyLoad imbalanceOOMKilledIntermittent connection failures

   Trigger

   Upgrading the HAProxy ingress controller to v0.6 and going back from Service virtual IPs to individual pod IPs as backends, on the understanding that the sort-backends flag introduced in v0.5, false by default, meant backends were randomly shuffled.

   Root cause

   Software defect

   Component

   Ingress controllerHAProxyConfiguration templating

   Contributing factor

   Unrepresentative stagingUnverified assumptionIgnored early signal

   Detection

   A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.

   Remediation

   • switched the affected service from pod IPs back to the Service virtual IP through a config change in the internal deployment tool Slingshot
   • patched the controller to keep each backend's servers in an insertion-ordered slice and changed the template to iterate over that slice
   • tested the patch in sandbox clusters and rolled it out to the production clusters

   Action items

   • Run the internal controller patch in production and return to pod IPs as the default backends
   • Raise the problem upstream with a GitHub issue and pull request
   • Test changes to cluster components against the specific behaviour and conditions that matter

   Lessons

   The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.

   View case

By root cause5