Cases24 cases

12 to 16 of 24

1. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents

   Palark2023

   Symptoms

   A tech support engineer called Palark to say that a customer's cluster, production environment included, had practically stopped working. On the control plane nodes the kube-apiserver processes had used up all the memory and were crashing and restarting over and over, so the Kubernetes API was unreachable and completely non-functional.

   Control plane unresponsiveOOMKilled

   Trigger

   Cilium agent pods on more than 200 nodes sent LIST requests to the Kubernetes API en masse at the same time.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverCilium

   Contributing factor

   Unreviewed defaults

   Detection

   A tech support engineer phoned the team to report that the customer's cluster was practically non-operational.

   Remediation

   • added CPU and memory to the control plane nodes in two steps, from 8 CPUs and 16 GB of RAM to 16 CPUs and 64 GB, until the API stabilised
   • restarted the Cilium agents in a test window agreed with the customer to reproduce the memory growth
   • deployed a FlowSchema and a PriorityLevelConfiguration named cilium-pods that limit and queue LIST requests from the Cilium service accounts to cilium.io resources
   • reduced the control plane nodes back to their original resources

   Action items

   • Make an API Priority and Fairness configuration a standard part of Palark's Kubernetes platform

   Lessons

   In a large cluster, many agents sending LIST requests at once can exhaust kube-apiserver memory, and with several API servers the crash moves from one to the next. APF can hold back such a client: a FlowSchema matches its requests and a PriorityLevelConfiguration gives them a small concurrency share and a queue, so the requests run more slowly instead of all at once. With APF enabled, the total concurrency it divides is the sum of --max-requests-inflight (default 400) and --max-mutating-requests-inflight (default 200), shared among priority levels in proportion to their assured concurrency shares, so raising one level's share lowers every other level's limit. Long-running requests such as logs and exec, and WATCH requests, are not subject to APF. The response headers X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID show where a request landed, and the metrics Apiserver_flowcontrol_rejected_requests_total, Apiserver_current_inqueue_requests and Apiserver_flowcontrol_request_execution_seconds, with the debug endpoints /debug/api_priority_and_fairness/dump_priority_levels and /debug/api_priority_and_fairness/dump_queues, show how the queues behave.

   View case
2. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout

   Grafana Labs2022

   Symptoms

   During a rollout of the Grafana Loki distributor Deployment in one Grafana Cloud Logs environment, the write path went down for about 25 minutes. The newly started distributor pods were OOM-killed over and over, never passed their health and readiness checks, and the rollout stopped making progress.

   Service outageOOMKilledLoad imbalanceProbe failures

   Trigger

   Step 1 of moving the Loki ring off a separately deployed key-value store onto embedded memberlist added a new label, gossip_ring_member: 'true', to the distributor pods and Service, together with a small config change, and the change required a rollout of the distributors.

   Root cause

   Configuration change

   Component

   Configuration templatingService discoveryContainer memory limit

   Contributing factor

   Unreviewed defaultsIgnored early signalUnrepresentative staging

   Detection

   The first clue was kubectl get pods and the kube_deployment_status_replicas_updated metric showing that only 16 of 140 distributor pods had been updated.

   Remediation

   • scaled the distributor Deployment out horizontally so the load spread over more pods with the new label and the rollout could continue
   • reverted the migration step to end the outage
   • set service_ignored_labels:: [$._config.gossip_member_label] and passed it to k.util.serviceFor so the label is no longer copied into the Service selector

   Action items

   • Update the function that generates a Service from a Deployment or StatefulSet to handle edge cases, including this one

   Lessons

   A Service whose selector is built by copying every label of a workload changes which pods it matches as soon as a new label is added. During a rollout it then matches only the new pods, which receive all the traffic before there are enough of them. A few errors during a rollout in other environments hid the problem, because those environments gave distributors more memory and could absorb the shift. Manifest diffs kept for every pull request made it possible to see that the Service selector had changed.

   View case
3. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary

   GitLab2022

   Symptoms

   On 2022-04-25 the Sidekiq Apdex (the share of background jobs that finish within a target time) dropped, and an engineer declared an incident in Slack at 15:13 UTC. Users saw intermittent slowness across GitLab.com, and asynchronous work such as pipeline creation and import and export took unusually long.

   High latencyDatabase overloadOOMKilledHigh error rate

   Trigger

   The Consul agent in GitLab.com's GKE pods had memory demand spikes that did not fit in the pods' 500MB memory limit, and the agent was killed again and again.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitService discoveryDatabase

   Contributing factor

   –

   Detection

   An engineer declared the incident in Slack at 15:13 UTC after the Sidekiq Apdex dropped; the second event was found when monitoring detected an elevated error rate at 01:21 UTC on 2022-04-26.

   Remediation

   • planned at 16:13 to raise the per-user connection limit for the gitlab database user and the pgbouncer backend pools for the main database; changes were applied at 17:26 and confirmed at 17:47
   • raised the pod memory limits from 500MB to 1GB so the Consul agents survive memory spikes

   Action items

   • Monitor the Consul agents on GKE pods to see whether the 1GB memory limit keeps them healthy
   • Investigate the Consul memory demand spikes and what can be done beyond raising the hard memory limit

   Lessons

   When the local Consul agent that Rails uses for database service discovery keeps dying, Rails does not fail; it sends every query to the last known primary. What engineers saw was database saturation and slow background jobs, which pointed the investigation first at a recent pgbouncer pool change. The memory limit of the Consul agent in every pod therefore decided the load on the primary database.

   View case
4. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case
5. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry

   GitHub2020

   Symptoms

   At 08:18 UTC on 13 July 2020, production Kubernetes pods began to be marked unavailable. The loss of pods spread through the clusters and reduced capacity until GitHub.com services went down.

   OOMKilledImage pull failuresService outage

   Trigger

   A container in the production pods exceeded its memory limit and was terminated; a routine DNS maintenance operation completed earlier had left the clusters unable to reach the image registry.

   Root cause

   DNS or network fault

   Component

   Container memory limitContainer registry and its credentials

   Contributing factor

   Overlapping failuresHarmful manual actionSingle point of failure

   Remediation

   • restarted the process with the cached DNS records so the clusters could reach the registry
   • fetched the container images, redeployed and recovered the services

   Action items

   • Improve monitoring so pod restarts cannot fail again in the same pattern
   • Minimise the dependency on the image registry
   • Expand validation during DNS changes
   • Re-evaluate all existing Kubernetes deployment policies
   • Improve progressive deployments so the impact of a deployment can be evaluated incrementally

   Lessons

   A container that production traffic does not need can still take pods out of service, because a pod is available only when all of its containers are healthy. ImagePullPolicy Always turns every container restart into a registry dependency, so an unrelated DNS change that breaks registry access turns a one-minute self-healing event into a long outage, and a mass redeploy spreads it.

   View case

By root cause5