Cases14 cases

9 to 13 of 14

1. IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accounts

   Linode2023

   Symptoms

   Resizing an LKE cluster produced an incorrect number of nodes, so some customer clusters were resized in unexpected ways and the extra worker nodes were billed to the customer accounts. The clusters themselves kept functioning. Linode's first status update, at 05:07 UTC on 1 June 2023, advised customers to avoid resizing their clusters if possible.

   Autoscaler misbehaviour

   Trigger

   At about 16:00 UTC on 31 May 2023 a change to the API that caused LKE clusters to scale improperly was pushed to production.

   Root cause

   Faulty deployment

   Component

   Node pool

   Contributing factor

   Gap in automated tests

   Detection

   The first status update, at 05:07 UTC on 1 June, said Linode had been made aware of an issue when resizing LKE clusters, about 13 hours after the change went out.

   Remediation

   • advised customers to avoid resizing their clusters
   • deployed a hotfix at 09:40 UTC on 1 June that corrected the scaling behaviour
   • left the extra worker nodes in place to avoid disrupting customer workloads
   • contacted affected customers by ticket about credit adjustments and further mitigation

   Action items

   • Audit the automated tests
   • Add pre-deployment checklist items so the same failure does not happen again

   Lessons

   An API change that alters scaling behaviour resized customer clusters and added cost while the clusters themselves kept working; Linode announced the issue about 13 hours after the change went out. Removing the extra nodes would have disrupted workloads, so the cost was handled through credits instead.

   View case
2. IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently

   Linode2022

   Symptoms

   From about 18:30 UTC on June 15, 2022, customers of the Linode Kubernetes Engine (LKE) who tried to upgrade a cluster or add nodes saw those operations fail silently, without an error. The status page opened an investigation of an LKE issue at 19:54 UTC, and the affected list covered LKE in eleven regions.

   Upgrade stall

   Trigger

   An incremental update pushed to the LKE production environment at about 18:30 UTC.

   Root cause

   Faulty deployment

   Component

   Node pool

   Contributing factor

   –

   Detection

   The status page began investigating at 19:54 UTC, 1 hour 24 minutes after the update.

   Remediation

   • Released a follow-up LKE update at about 21:00 UTC that contained the missing changes

   Action items

   • Add steps and automation to the update process

   View case
3. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com

   CircleCI2021

   Symptoms

   At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.

   Service outageResources deleted

   Trigger

   An update deployed to a test Kubernetes cluster also removed Traefik and Calico resources from CircleCI's primary production cluster.

   Root cause

   Faulty deployment

   Component

   Ingress controllerCalico

   Contributing factor

   Excessive blast radiusNo runbookMonitoring blind spot

   Detection

   CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.

   Remediation

   • restored Traefik in the production cluster
   • restored Calico in the production cluster, after which pods served customer requests again

   Action items

   • Improve network-level documentation and runbooks to cover every layer of the Kubernetes networks
   • Improve monitoring and alerting for the network infrastructure
   • Add guardrails that isolate the production and staging clusters so they cannot interfere with each other

   Lessons

   A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again. Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.

   View case
4. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error

   UK Ministry of Justice2020

   Symptoms

   At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.

   Deploys failing

   Trigger

   Version 0.1.0 of the team's ingress controller module started creating a ValidatingWebhookConfiguration resource, and one new 0.1.0 ingress controller in the live-1 cluster failed to be created because of AWS resource limits.

   Root cause

   Faulty deployment

   Component

   Admission webhookIngress controller

   Contributing factor

   Single point of failure

   Detection

   The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.

   Remediation

   • removed by hand all the additional admission controllers created by module version 0.1.0
   • removed the admission webhook from the ingress controller module and released version 0.1.1
   • upgraded every ingress module on version 0.1.0 to 0.1.1

   Lessons

   Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.

   View case
5. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic

   Airbnb2019

   Symptoms

   An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

   Probe failuresService outage

   Trigger

   A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

   Root cause

   Faulty deployment

   Component

   Telemetry agentService discovery

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

   Action items

   • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check

   Lessons

   Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

   View case

By root cause1