Cases22 cases

17 to 21 of 22

1. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404

   DFDS2021

   Symptoms

   At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.

   Service outageHigh error rateProbe failures

   Trigger

   The automated Kubernetes node rollover on 2021-09-06 restarted all the Traefik v2 pods, five days after a malformed IngressRoute manifest had been applied.

   Root cause

   Software defect

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Change with a delayed effectNo configuration testsExcessive blast radius

   Detection

   All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.

   Remediation

   • changed the ALB health check path, which made the targets healthy but did not stop the 404s
   • rolled back by hand to Traefik v1 as the main ingress controller
   • deleted the malformed IngressRoute by hand and confirmed the Traefik v2 bug
   • applied the changes upstream

   Action items

   • Upgrade Traefik to v2.5, where the bug is fixed, and verify that it removes the bug
   • Change the Traefik v2 health check path to something more robust
   • Set up an external liveness probe on the ingress controller
   • Consider workload segmentation to limit the blast zone
   • Look into monitoring the Traefik v2 API for rapid changes
   • Look into smoke testing during and after node rollover

   Lessons

   A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik. On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.

   View case
2. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable

   Enigmo2020

   Symptoms

   Two or three days after Enigmo created EKS managed node groups and moved its pods there from the old node groups, the cluster suddenly became unstable: pods restarted again and again or could not stop and stayed around indefinitely, node CPU and memory load rose high, and the nodes became unusable from the cluster.

   Pods stuck in TerminatingSustained high node CPUMemory pressureNode NotReadyUpgrade stall

   Trigger

   Enigmo created EKS managed node groups that used an AMI containing the bug and moved its pods onto them, as preparation for upgrading EKS from 1.15 to 1.16.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imageNode pool

   Contributing factor

   Ineffective rollbackChange with a delayed effect

   Remediation

   • Changed the managed node groups' AMI version to the fixed release
   • Ran the managed node group update through CloudFormation
   • Watched the pods across all namespaces during the update
   • Force-deleted every pod that stayed in Terminating with kubectl delete --grace-period=0 --force

   Action items

   • Check the latest releases of the cluster's modules quarterly or monthly and upgrade them regularly

   Lessons

   As long as a team uses Amazon EKS or Kubernetes, it has to keep track of upgrades of the cluster and of modules such as ingress controllers, because a release with breaking changes can be announced without warning after a long stable period. Checking releases and upgrading on a regular schedule keeps the effect of each upgrade on the service small.

   View case
3. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods

   Airbnb2020

   Symptoms

   A production service at Airbnb showed a high error rate and could not be deployed. The deploy failed with a cryptic Kubernetes error about being unable to access invalid index 5.

   High error rateThroughput degradationDeploys failing

   Trigger

   A change to the mutating admission controller, which had been rolled out within the previous month, had gone in seven days before the incident surfaced; when it had to delete two items from an array in a spec, it generated an invalid JSON patch that removed index 2 and then index 3.

   Root cause

   Software defect

   Component

   Admission webhookHorizontal pod autoscaler

   Contributing factor

   Change with a delayed effectMonitoring blind spotIgnored early signal

   Detection

   The affected service reported a high error rate and its deploy failed; the slow degradation of other services over the previous seven days had gone unnoticed.

   Remediation

   • deleted the mutating admission controller from the affected cluster

   Action items

   • Create alerts from the existing metrics that would have shown the problem earlier

   Lessons

   Keep in mind that a mutating admission controller exists in the cluster; a recently added one is easy to forget during an incident. Search for the exact error message early instead of working on the visible symptoms. Page a teammate when progress stalls.

   View case
4. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"

   Wikimedia Foundation2019

   Symptoms

   On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.

   Control plane unresponsiveService outageTLS certificate errors

   Trigger

   At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.

   Root cause

   Configuration change

   Component

   Kubernetes API serveretcdConfiguration management tool

   Contributing factor

   Change with a delayed effectHarmful manual actionNo runbook

   Detection

   Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.

   Remediation

   • Rebooted the Kubernetes master and the etcd cluster, which did not help
   • Took a backup of etcd and reduced the etcd cluster to one node to rule out raft election flaps
   • Brought etcd back to full capacity through a full disaster recovery from the backup
   • Swapped the API server's CA certificate for the one from the etcd servers, which restored services until Puppet reverted it
   • Placed the etcd Puppet CA certificate in /var/lib/puppet/client/ssl/certs/ca.pem so Puppet kept the right CA

   Action items

   • Document the etcd cluster processes
   • Upgrade Toolforge Kubernetes to improve etcd and Kubernetes error reporting and stop needing an API server restart to add a tool
   • Make kube2proxy more resilient to API server failures
   • Audit the Puppet CA certificates so the tools project has no more than one
   • Make the Kubernetes control plane highly available

   Lessons

   Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.

   View case
5. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods

   Pusher2019

   Symptoms

   During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

   Service outageHigh error rate

   Trigger

   A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

   Root cause

   Configuration change

   Component

   ConfigMaps and Secrets

   Contributing factor

   Change with a delayed effectUnverified assumption

   Action items

   • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately

   Lessons

   A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

   View case

By root cause6