Incident Library

Applied filters SymptomIntermittent timeoutsClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

5 results

Lit on the graph.

GitLab, 2022
Microsoft Azure, 2020
Grafana Labs, 2020
Omio, 2020
Airbnb, 2019

Cases21 cases

14 to 18 of 21

IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
GitLab2022
Symptoms
At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.
High error rate High latency Intermittent timeouts
Trigger
A k8s-workloads/gitlab-com pipeline deployed merge request !1643, a GitLab chart bump that changed the port names of Kubernetes Services and Deployments (related to the Prometheus service and pod monitors), starting with the canary environment gprd-cny at 07:49 UTC.
Root cause
Configuration change
Component
Helm Cloud load balancer HAProxy
Contributing factor
Automation amplification
Detection
A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.
Remediation
drained the canary environment and re-enabled it after twenty minutes
cancelled the deployment jobs for clusters c and d before they started
drained gprd-us-east1-d in HAProxy and re-enabled it five minutes later
raised the minimum replica count above 100 (merge request !1650)
drained gprd-us-east1-c in HAProxy, deployed the change to it, and re-enabled it
reverted the minimum replica increase (merge request !1652)
Action items
Implement graceful draining of a whole regional cluster
Create a Chatops command that puts PagerDuty into maintenance mode
Lessons
Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes. GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.
View case
IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNI
Microsoft Azure2020
Symptoms
In newly created AKS clusters using Azure CNI in Central India, kubectl exec and kubectl logs could time out, and pod-to-pod or pod-to-node traffic could time out. The cluster diagnostics page showed tunnel exec issues. Clusters using kubenet were not affected.
Intermittent timeouts
Trigger
VMs in Central India started being assigned MAC addresses in a much higher range than before (06 prefix rather than the usual 00).
Root cause
Software defect
Component
Azure CNI Node operating system image
Contributing factor
–
Remediation
told owners of affected scale-set clusters to run a node image upgrade, or any Kubernetes upgrade such as a patch version, to get the Azure CNI that does not use a bridge
told owners of availability-set clusters to scale out and delete the VMs with a high-range MAC address in bridge mode
advised recreating affected clusters that were brand new
Action items
Roll out a hotfix that makes transparent mode the default in all regions
Lessons
A change in how the platform assigns MAC addresses can wake a long-dormant bug in the network plugin. Knowing which CNI mode the nodes run (bridge or transparent) and keeping node images current decides whether such a bug reaches you.
View case
IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
Grafana Labs2020
Symptoms
About a minute after the etcd leader changed, two replicas of the Cortex distributor started logging "context deadline exceeded" when they tried to read keys from etcd. Writes from some customers that landed on those two replicas failed, while the other distributor replicas kept working, so the service saw an error rate of about 20% on writes to the London region of Hosted Prometheus.
High error rate Intermittent timeouts
Trigger
A scheduled Kubernetes version upgrade abruptly terminated the node that ran the etcd leader at 11:19:00 UTC.
Root cause
Software defect
Component
etcd Node pool
Contributing factor
Unreviewed defaults No safeguard on destructive change
Detection
An SLO-based alert paged the on-call engineer at 11:31, 11 minutes after the errors started. At a 20% error rate the monthly SLA would have taken about 18 hours to breach.
Remediation
Restarted the two affected Cortex distributor replicas at 11:32, which stopped the errors
Action items
Enable gRPC keepalive probes in the Cortex etcd client (DialKeepAliveTime 10s, DialKeepAliveTimeout, PermitWithoutStream)
Make the Kubernetes upgrade process terminate pods and machines gracefully, including the etcd pods managed by the etcd operator
Lessons
When a node dies abruptly, clients holding TCP connections to it are not told; without keepalive probes an etcd client can keep sending requests into a dead connection long after a new leader exists. A failover that works on the server side still needs clients that detect the dead peer. Node replacement during cluster upgrades should drain pods gracefully, including pods owned by operators that the upgrade scripts do not know about, and should give instances enough time to shut down.
View case
IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
Omio2020
Symptoms
For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.
CPU throttling Probe failures High error rate High latency Intermittent timeouts Intermittent connection failures
Trigger
Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.
Root cause
Software defect
Component
Container CPU limit Linux kernel kubelet
Contributing factor
Unknown limit Monitoring blind spot Unpatched known bug
Detection
Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.
Remediation
removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters
Action items
Deploy the kernel with the CFS quota bug fix across all clusters
Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected
Lessons
A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.
View case
IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
Airbnb2019
Symptoms
On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.
High latency Intermittent timeouts Load imbalance DNS resolution failures
Trigger
Several services sent a high volume of external DNS queries, all of which went through kube-dns.
Root cause
Capacity exhaustion
Component
kube-dns kube-proxy Client-side DNS resolver and cache
Contributing factor
Unreviewed defaults
Detection
Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.
Lessons
By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.
View case

Check your cluster for this

These 21 cases involve Linux kernel and kubelet. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap