Cases9 cases

6 to 9 of 9

1. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503

   DFDS2021

   Symptoms

   At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.

   Service outageHigh error rate

   Trigger

   The pipeline replaced the AWS VPC CNI as part of a refactoring that moved EKS add-ons from kubectl to native Terraform management.

   Root cause

   Configuration change

   Component

   AWS VPC CNICloud load balancerTerraform

   Contributing factor

   Unverified assumptionNo configuration tests

   Detection

   Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.

   Remediation

   • committed a fix that reverted the infrastructure-modules reference
   • re-triggered the pipeline after it failed on an AWS API timeout
   • let the pipeline reattach the target groups to the ALB

   Action items

   • Revert the ALB de-registration delay to the default of 300 seconds
   • Update the roll-over script to set the ALB de-registration delay to zero temporarily before a roll-over
   • Create an ingress test suite

   Lessons

   The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.

   View case
2. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out

   DFDS2020

   Symptoms

   At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

   Deploys failingControl plane lag

   Trigger

   During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

   Root cause

   Configuration change

   Component

   Admission webhookKubernetes API server

   Contributing factor

   No configuration testsUnverified assumptionMonitoring blind spot

   Detection

   Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

   Remediation

   • toggled the Alcide admission controller to verify that it caused the timeouts
   • permanently deleted the Alcide admission controller

   Action items

   • Monitor Kubernetes deployments for timeouts
   • Monitor 5xx status codes from the Kubernetes API server

   Lessons

   New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

   View case
3. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one

   Spotify2019

   Symptoms

   Terraform, applied from the shared cluster configuration repository after a merge, deleted Spotify's production GKE cluster in Asia. While the team tried to recreate it, Terraform deleted a second production cluster, so two of the three production clusters were gone.

   Resources deleted

   Trigger

   A feature team's pull request that added three users to an existing RBAC role binding file was merged before the cluster operators' pull request that imported the Asia cluster into Terraform state. The review build of the import pull request had already written the cluster into the shared state file, and the merged configuration had no definition of the Asia cluster, so Terraform removed it.

   Root cause

   Configuration change

   Component

   TerraformCI systemGKE control plane

   Contributing factor

   No safeguard on destructive changeMissed in reviewUnverified assumption

   Detection

   Internal service discovery could not reach the deleted cluster and the cluster operator team was paged.

   Remediation

   • removed the deleted clusters from the service discovery configuration and restarted service discovery so clients fell back to the instances outside Kubernetes
   • one team that ran half on Kubernetes and half on VMs created more VMs outside Kubernetes to handle the load
   • recreated the clusters, using the hourly backups
   • updated the places where the old clusters' master IPs were hardcoded

   Action items

   • Post the Terraform dry-run output as a comment on each pull request
   • Require a status check that the branch is up to date with master, so the dry run reflects what will actually happen
   • Require an approving review on pull requests
   • Fail reviews automatically when certain keywords such as destroy appear; cluster deletions are done manually outside this repository
   • Add linters and validators for the cluster configuration
   • Schedule and announce disaster recovery tests that cover a cluster going away and smaller failures

   Lessons

   Plan for failure from the start and migrate large infrastructure gradually, so there is redundancy and a way back. A backup whose restore has never been tested is not a backup. Practising disaster recovery paid off: restoring the first deleted cluster took three and a quarter hours, and the team can now restore a larger cluster in one hour. What matters is not who made the mistake but what was learned from it.

   View case
4. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods

   Pusher2019

   Symptoms

   During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

   Service outageHigh error rate

   Trigger

   A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

   Root cause

   Configuration change

   Component

   ConfigMaps and Secrets

   Contributing factor

   Change with a delayed effectUnverified assumption

   Action items

   • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately

   Lessons

   A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

   View case

By root cause1