Cases12 cases

6 to 10 of 12

1. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1

   DigitalOcean2024

   Symptoms

   At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

   Resources deletedControl plane unresponsive

   Trigger

   An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

   Root cause

   Configuration change

   Component

   etcdKubernetes API server

   Contributing factor

   Automation amplificationNo safeguard on destructive changeExcessive blast radius

   Detection

   Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

   Remediation

   • restored each affected cluster's etcd from its latest twice-daily backup
   • restored the stateless control plane components after etcd
   • patched the recovery mechanism after unexpected problems during recovery
   • found and fixed an additional small bug during recovery
   • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources

   Action items

   • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment

   Lessons

   When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

   View case
2. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic

   GitLab2022

   Symptoms

   GitLab's monitoring and alerting fired at once. One of the alerts reported a Patroni failover, and users of GitLab.com received 5XX errors; the CDN counted about 13 million requests answered with 503 during the outage.

   Service outageHigh error rateResources deleted

   Trigger

   An engineer ran a routine task with GitLab's tools in a local development environment; during a local deploy or rollback operation the tool's context changed from local development to production, and the change was applied to production.

   Root cause

   Operator error

   Component

   Application database

   Contributing factor

   No safeguard on destructive change

   Detection

   GitLab's monitoring and alerting fired immediately; one of the alerts reported a Patroni failover.

   Remediation

   • re-applied the Consul release to the production Kubernetes cluster, with manual steps because the tooling did not support the redeploy directly

   Action items

   • Improve the Kubernetes deployment process to reduce the risk of running harmful deployment operations against production from local environments (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16875)
   • Integrate Teleport with the Kubernetes clusters, which could give SREs a read-only role by default (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16221)

   Lessons

   A tool that can switch its target from a local environment to production during a deploy or rollback turns routine local work into a production change. GitLab plans to improve its Kubernetes deployment process so that harmful deployment operations are less likely to be run against production from local environments, and to integrate Teleport with its clusters, which could give SREs a read-only role by default. Detection and diagnosis were fast because the alerts led straight from Patroni to its dependency, Consul. Bringing the deleted release back needed manual work, because the tooling did not make the redeploy straightforward, but it was still done relatively quickly.

   View case
3. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment

   InfluxData2022

   Symptoms

   Right after a pull request to the infrastructure-as-code repository was merged, every API endpoint of one InfluxDB Cloud production cluster began returning 404 errors. Writes and reads failed alike: no one could collect data, scheduled tasks failed to run and external queries did not work.

   Service outageResources deletedHigh error rate

   Trigger

   A merged pull request whose rendered YAML added a new Argo CD Application and AppProject that an automation error had named app01-cluster01, the name of the existing production Application, instead of app02-cluster01; reviewers did not notice the app01 and app02 difference.

   Root cause

   Configuration change

   Component

   Argo CDPersistent volumes and stateful setsConfiguration templating

   Contributing factor

   Missed in reviewNo runbookNo safeguard on destructive changeMonitoring blind spotAutomation amplification

   Detection

   Monitoring and alerting did not detect the issue right away, so the team's first response was to try to roll the change back rather than plan a recovery. The write-up does not say how the team found out.

   Remediation

   • stopped the revert of the merged pull request
   • restored the etcd metadata cluster
   • recreated the missing Kubernetes objects so the retained PersistentVolumes became available, then recreated the StatefulSets
   • restored the Zookeeper volumes from an hourly Velero backup
   • rebuilt the Kafka StatefulSet one pod at a time with health and readiness checks disabled
   • accepted writes into the Kafka write-ahead log and scaled up the write path with more replicas and higher memory requests and limits
   • recreated the storage pods, which downloaded data from object storage, reindexed it and replayed the write-ahead log
   • re-enabled Argo CD, queries, dashboards and scheduled tasks, restored replica counts and ran final QA checks

   Action items

   • Write runbooks for restoring state when Kubernetes objects such as PersistentVolumeClaims are deleted but the data on the underlying volumes is preserved
   • Set every volume in every environment to retain data when its PersistentVolumeClaim is deleted
   • Improve the process for handling public-facing incidents
   • Store generated YAML as one object per file, so an overwrite of an existing object shows up as such in review
   • Detect duplicate objects when generating YAML and warn before review, treating objects with the same group, namespace and name as the same even when the apiVersion string differs
   • Add the Argo CD Prune=false annotation to all stateful resources and to Namespace objects managed by Argo CD
   • Set FailOnSharedResource=true on Argo CD Applications so Argo CD fails before changing an object another Application manages or managed

   Lessons

   The biggest gap was that monitoring did not notice the deletion quickly, which pushed the team toward a reflex rollback; in a stateful system a revert can create a new, empty instance instead of bringing the old one back. Retained volumes and clients that retry writes are what kept data loss at zero. Rendering many objects into one file let an overwrite of an existing object look like an addition, so generated configuration needs one object per file and duplicate detection that compares objects the way Kubernetes does. Stateful resources need protection from pruning, and that protection must cover their Namespace too, or deleting the namespace removes everything inside it.

   View case
4. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded

   OpenAI2022

   Symptoms

   At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.

   Service outageHigh error rateIntermittent timeouts

   Trigger

   A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.

   Root cause

   Operator error

   Component

   Ingress controllerModel hub

   Contributing factor

   No safeguard on destructive changeNo headroomDeferred upgradesOverlapping failures

   Remediation

   • re-allocated the models from the lost cluster to the remaining two clusters
   • rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
   • built workarounds with Azure support after losing the connection to Azure Private Link
   • tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
   • moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT

   Action items

   • Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
   • Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
   • Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
   • Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
   • Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
   • Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
   • Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
   • Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
   • Change how model weights are stored so they are more efficient and faster to load

   Lessons

   Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.

   View case
5. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns

   UK Ministry of Justice2020

   Symptoms

   At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

   Service outageResources deletedMonitoring outageStale endpoint routing

   Trigger

   A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

   Root cause

   Operator error

   Component

   Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus

   Contributing factor

   No safeguard on destructive change

   Detection

   The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

   Remediation

   • restored the default ingress controller together with external-dns and kiam, which it depends on
   • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
   • restored the remaining components with a targeted apply
   • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository

   View case

By root cause4