Cases2 cases

1 to 2 of 2

1. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry

   GitHub2020

   Symptoms

   At 08:18 UTC on 13 July 2020, production Kubernetes pods began to be marked unavailable. The loss of pods spread through the clusters and reduced capacity until GitHub.com services went down.

   OOMKilledImage pull failuresService outage

   Trigger

   A container in the production pods exceeded its memory limit and was terminated; a routine DNS maintenance operation completed earlier had left the clusters unable to reach the image registry.

   Root cause

   DNS or network fault

   Component

   Container memory limitContainer registry and its credentials

   Contributing factor

   Overlapping failuresHarmful manual actionSingle point of failure

   Remediation

   • restarted the process with the cached DNS records so the clusters could reach the registry
   • fetched the container images, redeployed and recovered the services

   Action items

   • Improve monitoring so pod restarts cannot fail again in the same pattern
   • Minimise the dependency on the image registry
   • Expand validation during DNS changes
   • Re-evaluate all existing Kubernetes deployment policies
   • Improve progressive deployments so the impact of a deployment can be evaluated incrementally

   Lessons

   A container that production traffic does not need can still take pods out of service, because a pod is available only when all of its containers are healthy. ImagePullPolicy Always turns every container restart into a registry dependency, so an unrelated DNS change that breaks registry access turns a one-minute self-healing event into a long outage, and a mass redeploy spreads it.

   View case
2. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances

   Datadog2019

   Symptoms

   Users reported that their jobs were not starting, with some sign that image pulls were involved. Many pods across the cluster were in error states related to image pulls, while applications that were already running were not affected.

   Image pull failuresService outage

   Trigger

   A change to the permissions on a storage bucket that one application needed to read. The application could no longer start, and its DaemonSet, running on roughly a thousand nodes of the cluster, went into a crash loop with imagePullPolicy: Always, so every restart pulled the image again although it was already on the node.

   Root cause

   Capacity exhaustion

   Component

   Container registry and its credentialsNAT gatewayApplication databaseAdmission webhook

   Contributing factor

   Automation amplificationSingle point of failureUnknown limitHarmful manual actionExcessive blast radius

   Detection

   Users reported that their jobs were not starting; monitoring had not caught it first.

   Remediation

   • replaced the affected NAT instances so registry traffic came from new source IPs
   • reconfigured the static IPs and firewall rules so applications could reach the Cloud SQL database again

   Action items

   • Add an admission webhook that rejects floating image tags, starting with latest
   • Apply the image tag webhook to workload objects only, not to pods, after it blocked pods created by in-cluster controllers

   Lessons

   The speakers' takeaway is to be very careful with DaemonSets in large clusters: they say they had broken everything DaemonSets depend on, including DNS, Vault and the image repository, and that it is very easy to break things when managing them by the thousand. imagePullPolicy: Always made every crash-loop restart pull the image again. Sending all registry traffic through three NAT instances put the whole region behind three source IPs, which the registry rate-limited, and changing those IPs cut the connections to a critical database reached over the internet through them. An admission webhook on pods rejects the pods that controllers in the cluster create, where no user sees the error.

   View case

By root cause2