Graph · Incident Library

Incident Library

Root cause
Component
Symptom
Contributing factor
Case

5 results

Lit on the graph.

DFDS, 2022
Lotte E-commerce, 2022
UK Ministry of Justice, 2022
OpenAI, 2022
GitLab, 2022

Cases26 cases

14 to 18 of 26

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
DFDS2022
Symptoms
On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.
Node NotReady Upgrade stall Intermittent timeouts Intermittent connection failures
Trigger
At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.
Root cause
Software defect
Component
Node operating system image Linux kernel kubelet AWS VPC CNI Node launch template Terraform
Contributing factor
No staged rollout Automation amplification Overlapping failures Monitoring blind spot No health probes
Detection
The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.
Remediation
cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
changed the Terraform infrastructure-modules so a specific AMI could be set
rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline
Action items
Reach out to teams with suggestions about readiness and liveness probes
Create a dashboard for readiness and liveness probe coverage
Write a wiki article on readiness and liveness probes and their usefulness
Modify infrastructure-modules to allow specifying a particular AMI
Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
Implement alerting on abnormal NodeNotReady events and on increased socket use
Create a learning path for Kubernetes within the team
Collect logs from nodes centrally
Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts
Lessons
When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.
View case
IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
Lotte E-commerce2022
Symptoms
LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.
High error rate Intermittent timeouts Stale endpoint routing
Trigger
HPA scale-in deleting pods, and the deletion of Blue pods after a blue/green switch of the Service label selector.
Root cause
DNS or network fault
Component
Horizontal pod autoscaler Pod termination kube-proxy conntrack table Cloud load balancer Ingress controller
Contributing factor
Unreviewed defaults
Remediation
delayed container shutdown on pod deletion (a wait in the SIGTERM handler or a preStop sleep, capped by terminationGracePeriodSeconds) so containers stop only after the pod has left the Service endpoints
switched some workloads' Ingress-created ALBs from instance mode to IP mode so traffic goes straight to pod IPs without iptables and conntrack
Action items
Move all workloads to ALB IP mode gradually
Lessons
Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application. With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.
View case
IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone
UK Ministry of Justice2022
Symptoms
On 11 July 2022 users reported in the #ask-cloud-platform Slack channel that their applications on the Ministry of Justice Cloud Platform were slow some of the time. About 25% of ingress traffic was affected: after an AWS incident the day before, one of the three network interfaces of the 'default' ingress controllers performed slowly, and clients timed out when they tried to connect to one of the Elastic IPs of the Network Load Balancer (NLB) in front of them.
High latency Intermittent timeouts
Trigger
Some instances of the AWS NLB health checking subsystem were restarted, which set off a latent software race condition in that subsystem.
Root cause
Cloud provider outage
Component
Cloud load balancer Ingress controller
Contributing factor
–
Detection
Users reported slow performance of their applications in the #ask-cloud-platform Slack channel.
Remediation
AWS restarted the NLB health checking subsystem, which refreshed its list of targets and recovered the NLB in the affected Availability Zone
Action items
Raised mitigation tickets after a post-incident review, filed as post-aws-incident issues in the ministryofjustice/cloud-platform repository
View case
IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
OpenAI2022
Symptoms
At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.
Service outage High error rate Intermittent timeouts
Trigger
A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.
Root cause
Operator error
Component
Ingress controller Model hub
Contributing factor
No safeguard on destructive change No headroom Deferred upgrades Overlapping failures
Remediation
re-allocated the models from the lost cluster to the remaining two clusters
rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
built workarounds with Azure support after losing the connection to Azure Private Link
tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT
Action items
Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
Change how model weights are stored so they are more efficient and faster to load
Lessons
Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.
View case
IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
GitLab2022
Symptoms
At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.
High error rate High latency Intermittent timeouts
Trigger
A k8s-workloads/gitlab-com pipeline deployed merge request !1643, a GitLab chart bump that changed the port names of Kubernetes Services and Deployments (related to the Prometheus service and pod monitors), starting with the canary environment gprd-cny at 07:49 UTC.
Root cause
Configuration change
Component
Helm Cloud load balancer HAProxy
Contributing factor
Automation amplification
Detection
A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.
Remediation
drained the canary environment and re-enabled it after twenty minutes
cancelled the deployment jobs for clusters c and d before they started
drained gprd-us-east1-d in HAProxy and re-enabled it five minutes later
raised the minimum replica count above 100 (merge request !1650)
drained gprd-us-east1-c in HAProxy, deployed the change to it, and re-enabled it
reverted the minimum replica increase (merge request !1652)
Action items
Implement graceful draining of a whole regional cluster
Create a Chatops command that puts PagerDuty into maintenance mode
Lessons
Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes. GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.
View case

Check your cluster for this

These 26 cases involve Linux kernel and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap