Cases11 cases

4 to 8 of 11

1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services

   OpenAI2024

   Symptoms

   Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

   Service outageHigh error rateControl plane unresponsiveDNS resolution failures

   Trigger

   At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

   Root cause

   Configuration change

   Component

   Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access

   Detection

   OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

   Remediation

   • Scaled clusters down to reduce the total Kubernetes API load
   • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
   • Scaled up the Kubernetes API servers
   • Removed the telemetry service once enough control was back
   • Moved traffic from impacted clusters to healthy ones
   • Intervened manually in clusters where simultaneous resource downloads saturated resource limits

   Action items

   • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
   • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
   • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
   • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
   • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster

   Lessons

   Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

   View case
2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters

   OpenAI2024

   Symptoms

   From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

   High error rateHigh latencyControl plane unresponsive

   Trigger

   A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

   Root cause

   Configuration change

   Component

   Kubernetes API server

   Contributing factor

   Excessive blast radiusLimited debug access

   Remediation

   • moved workloads out of the impacted clusters by hand
   • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover

   Action items

   • Lock namespace changes so this incident cannot recur
   • Freeze certain types of infrastructure deploys
   • Allowlist identities to prevent a recurrence of this class of incidents
   • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
   • Improve change management for fleet-wide infrastructure changes

   View case
3. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment

   InfluxData2022

   Symptoms

   Right after a pull request to the infrastructure-as-code repository was merged, every API endpoint of one InfluxDB Cloud production cluster began returning 404 errors. Writes and reads failed alike: no one could collect data, scheduled tasks failed to run and external queries did not work.

   Service outageResources deletedHigh error rate

   Trigger

   A merged pull request whose rendered YAML added a new Argo CD Application and AppProject that an automation error had named app01-cluster01, the name of the existing production Application, instead of app02-cluster01; reviewers did not notice the app01 and app02 difference.

   Root cause

   Configuration change

   Component

   Argo CDPersistent volumes and stateful setsConfiguration templating

   Contributing factor

   Missed in reviewNo runbookNo safeguard on destructive changeMonitoring blind spotAutomation amplification

   Detection

   Monitoring and alerting did not detect the issue right away, so the team's first response was to try to roll the change back rather than plan a recovery. The write-up does not say how the team found out.

   Remediation

   • stopped the revert of the merged pull request
   • restored the etcd metadata cluster
   • recreated the missing Kubernetes objects so the retained PersistentVolumes became available, then recreated the StatefulSets
   • restored the Zookeeper volumes from an hourly Velero backup
   • rebuilt the Kafka StatefulSet one pod at a time with health and readiness checks disabled
   • accepted writes into the Kafka write-ahead log and scaled up the write path with more replicas and higher memory requests and limits
   • recreated the storage pods, which downloaded data from object storage, reindexed it and replayed the write-ahead log
   • re-enabled Argo CD, queries, dashboards and scheduled tasks, restored replica counts and ran final QA checks

   Action items

   • Write runbooks for restoring state when Kubernetes objects such as PersistentVolumeClaims are deleted but the data on the underlying volumes is preserved
   • Set every volume in every environment to retain data when its PersistentVolumeClaim is deleted
   • Improve the process for handling public-facing incidents
   • Store generated YAML as one object per file, so an overwrite of an existing object shows up as such in review
   • Detect duplicate objects when generating YAML and warn before review, treating objects with the same group, namespace and name as the same even when the apiVersion string differs
   • Add the Argo CD Prune=false annotation to all stateful resources and to Namespace objects managed by Argo CD
   • Set FailOnSharedResource=true on Argo CD Applications so Argo CD fails before changing an object another Application manages or managed

   Lessons

   The biggest gap was that monitoring did not notice the deletion quickly, which pushed the team toward a reflex rollback; in a stateful system a revert can create a new, empty instance instead of bringing the old one back. Retained volumes and clients that retry writes are what kept data loss at zero. Rendering many objects into one file let an overwrite of an existing object look like an addition, so generated configuration needs one object per file and duplicate detection that compares objects the way Kubernetes does. Stateful resources need protection from pruning, and that protection must cover their Namespace too, or deleting the namespace removes everything inside it.

   View case
4. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors

   GitLab2022

   Symptoms

   At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.

   High error rateHigh latencyIntermittent timeouts

   Trigger

   A k8s-workloads/gitlab-com pipeline deployed merge request !1643, a GitLab chart bump that changed the port names of Kubernetes Services and Deployments (related to the Prometheus service and pod monitors), starting with the canary environment gprd-cny at 07:49 UTC.

   Root cause

   Configuration change

   Component

   HelmCloud load balancerHAProxy

   Contributing factor

   Automation amplification

   Detection

   A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.

   Remediation

   • drained the canary environment and re-enabled it after twenty minutes
   • cancelled the deployment jobs for clusters c and d before they started
   • drained gprd-us-east1-d in HAProxy and re-enabled it five minutes later
   • raised the minimum replica count above 100 (merge request !1650)
   • drained gprd-us-east1-c in HAProxy, deployed the change to it, and re-enabled it
   • reverted the minimum replica increase (merge request !1652)

   Action items

   • Implement graceful draining of a whole regional cluster
   • Create a Chatops command that puts PagerDuty into maintenance mode

   Lessons

   Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes. GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.

   View case
5. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503

   DFDS2021

   Symptoms

   At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.

   Service outageHigh error rate

   Trigger

   The pipeline replaced the AWS VPC CNI as part of a refactoring that moved EKS add-ons from kubectl to native Terraform management.

   Root cause

   Configuration change

   Component

   AWS VPC CNICloud load balancerTerraform

   Contributing factor

   Unverified assumptionNo configuration tests

   Detection

   Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.

   Remediation

   • committed a fix that reverted the infrastructure-modules reference
   • re-triggered the pipeline after it failed on an AWS API timeout
   • let the pipeline reattach the target groups to the ALB

   Action items

   • Revert the ALB de-registration delay to the default of 300 seconds
   • Update the roll-over script to set the ALB de-registration delay to zero temporarily before a roll-over
   • Create an ingress test suite

   Lessons

   The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.

   View case

By root cause1