Cases35 cases

16 to 20 of 35

1. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503

   DFDS2021

   Symptoms

   At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.

   Service outageHigh error rate

   Trigger

   The pipeline replaced the AWS VPC CNI as part of a refactoring that moved EKS add-ons from kubectl to native Terraform management.

   Root cause

   Configuration change

   Component

   AWS VPC CNICloud load balancerTerraform

   Contributing factor

   Unverified assumptionNo configuration tests

   Detection

   Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.

   Remediation

   • committed a fix that reverted the infrastructure-modules reference
   • re-triggered the pipeline after it failed on an AWS API timeout
   • let the pipeline reattach the target groups to the ALB

   Action items

   • Revert the ALB de-registration delay to the default of 300 seconds
   • Update the roll-over script to set the ALB de-registration delay to zero temporarily before a roll-over
   • Create an ingress test suite

   Lessons

   The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.

   View case
2. IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0

   Palark2021

   Symptoms

   After the last old Elasticsearch pod, elasticsearch-0, was deleted and came back on a new, empty persistent volume, `_cat/nodes` run inside that pod listed a single node: elasticsearch-0 itself, marked as master. The same command run inside elasticsearch-1 listed the other five nodes, elasticsearch-1 to elasticsearch-5, with elasticsearch-2 as master.

   Split brain

   Trigger

   Deleting the elasticsearch-0 pod and its PVC so that it restarted on an empty volume of the new StorageClass.

   Root cause

   Operator error

   Component

   Elasticsearch or OpenSearchPersistent volumes and stateful setsHelm

   Contributing factor

   Unverified assumption

   Detection

   The engineer running the migration checked `_cat/nodes` after elasticsearch-0 restarted and saw it alone in a cluster of its own.

   Remediation

   • scaled the three deployments of workers writing to Elasticsearch down to zero
   • listed elasticsearch-0, elasticsearch-1 and elasticsearch-2 under initial_master_nodes in the elasticsearch ConfigMap and removed the cluster.initial_master_nodes environment variable from the StatefulSet
   • saved and deleted the elasticsearch discovery Service and deleted the PVC of elasticsearch-0 while the rolling update ran
   • set initial_master_nodes to elasticsearch-3, elasticsearch-4 and elasticsearch-5, created an empty PVC for elasticsearch-0 and deleted all six pods
   • recreated the discovery Service from the saved manifest, after which the six nodes formed one cluster with every index green

   Action items

   • Restore the original initial_master_nodes values in the ConfigMap and restart all pods again
   • Move all shards to nodes 0, 1 and 2 and scale the cluster down from six nodes to three
   • Commit all the manual changes to the repository

   Lessons

   A setting that only matters when a cluster first starts, such as Elasticsearch's cluster.initial_master_nodes, can sit in a Helm chart template rather than in the ConfigMap that was reviewed. When a pod named in it starts with an empty data directory, it can bootstrap a new cluster beside the existing one. The author's conclusion is that a migration plan like this one breaks on any surprise and needs a plan B ready. Before running it in a more important environment, rehearse it in a test environment configured like production, and either schedule downtime or move the load to another cluster: stop the writers, take a fresh backup, then move the data.

   View case
3. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured

   Devsisters2021

   Symptoms

   During a team dinner, a Slack alert reported that one node of the CockroachDB cluster behind Cookie Run: Kingdom had gone down. The DevOps engineers were not worried at first, because CockroachDB is built to survive the loss of some nodes.

   Node NotReadyData lossService outage

   Trigger

   A cooling system failure in an AWS Tokyo region data centre overheated the servers in one availability zone, which held CockroachDB nodes.

   Root cause

   Cloud provider outage

   Component

   Application databaseHelm

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A Slack alert reported that a database node had gone down.

   Remediation

   • copied the data outside the broken ranges into new tables with INSERT INTO ... SELECT queries that skipped the affected ranges
   • ran debug unsafe-remove-dead-replicas together with Cockroach Labs engineers, recovering one of the two lost data ranges
   • restored the last data range with the storage-layer recovery method from the previous incident

   Action items

   • Configure CockroachDB Locality so data is spread evenly across availability zones
   • Deploy with one chart and a separate values file per availability zone to set Locality
   • Replace nodes by adding Locality-configured nodes, confirming replicas in every zone, then retiring the old nodes

   Lessons

   A replication factor sized for random instance retirements does not protect against a whole availability zone failing; for a game of this scale a multi-AZ placement is required, not optional. Keeping one uniform Helm chart made zone topology hard to set, which is part of why it was left out. A repetitive chart with a values file per zone sets it reliably.

   View case
4. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out

   DFDS2020

   Symptoms

   At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

   Deploys failingControl plane lag

   Trigger

   During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

   Root cause

   Configuration change

   Component

   Admission webhookKubernetes API server

   Contributing factor

   No configuration testsUnverified assumptionMonitoring blind spot

   Detection

   Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

   Remediation

   • toggled the Alcide admission controller to verify that it caused the timeouts
   • permanently deleted the Alcide admission controller

   Action items

   • Monitor Kubernetes deployments for timeouts
   • Monitor 5xx status codes from the Kubernetes API server

   Lessons

   New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

   View case
5. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub

   2i2c2020

   Symptoms

   At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.

   Service outageSustained high node CPU

   Trigger

   WER students' pods used a good chunk of their memory limit (tests had shown they almost always use just under 1G of their 2G limit, against a 512M guarantee) on nodes shared with the hub's core pods.

   Root cause

   Capacity exhaustion

   Component

   Node poolContainer memory limit

   Contributing factor

   No resource requests or limitsUnverified assumptionExcessive blast radius

   Detection

   WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).

   Remediation

   • removed memory overcommit for the WER students
   • raised resource guarantees for all core pods, which restarted them and moved some to a new node

   Action items

   • Put user pods in a separate pool so they do not create pressure on the core pods (infrastructure#89)
   • Set limits on the support infrastructure (Prometheus, Grafana, ingress) as well (infrastructure#90)
   • Document and think about overcommit ratios for memory usage (infrastructure#91)
   • Set up better Grafana dashboards to monitor resource usage (infrastructure#92)
   • Document how people can get kubectl access to the cluster so others can look into issues too (infrastructure#87)

   Lessons

   Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.

   View case

By root cause10