Cases33 cases

21 to 25 of 33

1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error

   UK Ministry of Justice2020

   Symptoms

   At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.

   Deploys failing

   Trigger

   Version 0.1.0 of the team's ingress controller module started creating a ValidatingWebhookConfiguration resource, and one new 0.1.0 ingress controller in the live-1 cluster failed to be created because of AWS resource limits.

   Root cause

   Faulty deployment

   Component

   Admission webhookIngress controller

   Contributing factor

   Single point of failure

   Detection

   The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.

   Remediation

   • removed by hand all the additional admission controllers created by module version 0.1.0
   • removed the admission webhook from the ingress controller module and released version 0.1.1
   • upgraded every ingress module on version 0.1.0 to 0.1.1

   Lessons

   Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.

   View case
2. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory

   UK Ministry of Justice2020

   Symptoms

   At 10:43 on 14 August 2020 two of the six ingress controller pods in the cluster were in a crash loop. Restarting those pods did not fix them.

   Container crash loop

   Root cause

   Capacity exhaustion

   Component

   Ingress controller

   Contributing factor

   Harmful manual actionSingle point of failure

   Detection

   Two of the six ingress controller pods were seen crash-looping at 10:43; the incident was declared at 11:01.

   Remediation

   • restarted the leader ingress controller pod so the other replicas could connect and get the latest nginx configuration

   View case
3. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry

   GitHub2020

   Symptoms

   At 08:18 UTC on 13 July 2020, production Kubernetes pods began to be marked unavailable. The loss of pods spread through the clusters and reduced capacity until GitHub.com services went down.

   OOMKilledImage pull failuresService outage

   Trigger

   A container in the production pods exceeded its memory limit and was terminated; a routine DNS maintenance operation completed earlier had left the clusters unable to reach the image registry.

   Root cause

   DNS or network fault

   Component

   Container memory limitContainer registry and its credentials

   Contributing factor

   Overlapping failuresHarmful manual actionSingle point of failure

   Remediation

   • restarted the process with the cached DNS records so the clusters could reach the registry
   • fetched the container images, redeployed and recovered the services

   Action items

   • Improve monitoring so pod restarts cannot fail again in the same pattern
   • Minimise the dependency on the image registry
   • Expand validation during DNS changes
   • Re-evaluate all existing Kubernetes deployment policies
   • Improve progressive deployments so the impact of a deployment can be evaluated incrementally

   Lessons

   A container that production traffic does not need can still take pods out of service, because a pod is available only when all of its containers are healthy. ImagePullPolicy Always turns every container restart into a registry dependency, so an unrelated DNS change that breaks registry access turns a one-minute self-healing event into a long outage, and a mass redeploy spreads it.

   View case
4. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data

   Palark2020

   Symptoms

   Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.

   Data lossService outage

   Trigger

   A RedisFailover change that raised the number of Redis replicas from one to two and, in the same rollout, lowered the containers' memory requests and limits from 22 GB to 19 GB.

   Root cause

   Software defect

   Component

   DatabasePersistent volumes and stateful setsContainer memory limit

   Contributing factor

   No staged rolloutSingle point of failureUnverified assumption

   Remediation

   • restored the Redis data from the backup prepared before the rollout

   Action items

   • Report the readiness probe flaw to the Redis Operator project and test the developers' fix
   • Reconsider the use of the Spotahome Redis Operator
   • Clean up the Redis databases and fix the application code that created the excess keys
   • Consider Redis built-in clustering and sharding

   Lessons

   Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty. A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.

   View case
5. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit

   Zalando2019

   Symptoms

   Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

   High error rateCredentials unavailableControl plane lag

   Trigger

   An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

   Root cause

   Capacity exhaustion

   Component

   kubeletCredential broker for podsKubernetes API serverCI system

   Contributing factor

   Unreviewed defaultsSingle point of failure

   Remediation

   • Scaled the CDP cluster up by hand so builder pods spread over more than one node

   Action items

   • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration

   View case

By root cause11