Cases11 cases

4 to 8 of 11

1. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes

   Reddit2023

   Symptoms

   About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

   Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag

   Trigger

   An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

   Root cause

   Breaking change in an upgrade

   Component

   CalicoKubernetes API serverAdmission webhooketcd

   Contributing factor

   Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure

   Detection

   The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

   Remediation

   • deleted the Open Policy Agent webhook configurations to stop API server timeouts
   • terminated all worker nodes of the cluster
   • restored the control plane from the pre-upgrade backup onto the first control plane node
   • switched off the cluster autoscaler until the control plane was stable
   • generated new TLS certificates so replacement control plane nodes could join etcd
   • brought traffic back in steps from 1% to 100%
   • restarted legacy services that had been stopped by hand

   Action items

   • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
   • Standardise components across clusters and replace bespoke configurations
   • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
   • Add pre-production clusters that can be tested with confidence
   • Improve the backup restore process and make cluster upgrades safe

   Lessons

   Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

   View case
2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes

   DFDS2022

   Symptoms

   At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

   Deploys failingContainer crash loop

   Trigger

   A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

   Root cause

   Certificate or credential expiry

   Component

   Policy engineAdmission webhook

   Contributing factor

   Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal

   Detection

   An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

   Remediation

   • reproduced the issue in a sandbox cluster and deployed the fix there first
   • updated Kyverno to a newer Helm chart version in the affected production cluster

   Action items

   • Implement monitoring on Kyverno
   • Create internal documentation on the Kyverno setup
   • Investigate the exact certificate issue in the third-party application
   • Consider changing the failurePolicy on ClusterPolicy resources going forward

   Lessons

   Operational issues should be added to the 'Ops Day to Day' board from the start.

   View case
3. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change

   UK Ministry of Justice2021

   Symptoms

   At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

   Deploys failing

   Trigger

   The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

   Root cause

   Breaking change in an upgrade

   Component

   Admission webhookIngress controller

   Contributing factor

   Unrepresentative staging

   Detection

   A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

   Remediation

   • rolled back the ModSec ingress controller module to v0.0.7

   Action items

   • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
   • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
   • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
   • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)

   View case
4. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods

   Airbnb2020

   Symptoms

   A production service at Airbnb showed a high error rate and could not be deployed. The deploy failed with a cryptic Kubernetes error about being unable to access invalid index 5.

   High error rateThroughput degradationDeploys failing

   Trigger

   A change to the mutating admission controller, which had been rolled out within the previous month, had gone in seven days before the incident surfaced; when it had to delete two items from an array in a spec, it generated an invalid JSON patch that removed index 2 and then index 3.

   Root cause

   Software defect

   Component

   Admission webhookHorizontal pod autoscaler

   Contributing factor

   Change with a delayed effectMonitoring blind spotIgnored early signal

   Detection

   The affected service reported a high error rate and its deploy failed; the slow degradation of other services over the previous seven days had gone unnoticed.

   Remediation

   • deleted the mutating admission controller from the affected cluster

   Action items

   • Create alerts from the existing metrics that would have shown the problem earlier

   Lessons

   Keep in mind that a mutating admission controller exists in the cluster; a recently added one is easy to forget during an incident. Search for the exact error message early instead of working on the visible symptoms. Page a teammate when progress stalls.

   View case
5. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out

   DFDS2020

   Symptoms

   At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

   Deploys failingControl plane lag

   Trigger

   During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

   Root cause

   Configuration change

   Component

   Admission webhookKubernetes API server

   Contributing factor

   No configuration testsUnverified assumptionMonitoring blind spot

   Detection

   Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

   Remediation

   • toggled the Alcide admission controller to verify that it caused the timeouts
   • permanently deleted the Alcide admission controller

   Action items

   • Monitor Kubernetes deployments for timeouts
   • Monitor 5xx status codes from the Kubernetes API server

   Lessons

   New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

   View case

By root cause7