Incident Library

Applied filters SymptomIntermittent timeoutsDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

5 results

Lit on the graph.

UK Ministry of Justice, 2022
OpenAI, 2022
GitLab, 2022
Chatwork, 2021
Microsoft Azure, 2020

Cases26 cases

16 to 20 of 26

IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone
UK Ministry of Justice2022
Symptoms
On 11 July 2022 users reported in the #ask-cloud-platform Slack channel that their applications on the Ministry of Justice Cloud Platform were slow some of the time. About 25% of ingress traffic was affected: after an AWS incident the day before, one of the three network interfaces of the 'default' ingress controllers performed slowly, and clients timed out when they tried to connect to one of the Elastic IPs of the Network Load Balancer (NLB) in front of them.
High latency Intermittent timeouts
Trigger
Some instances of the AWS NLB health checking subsystem were restarted, which set off a latent software race condition in that subsystem.
Root cause
Cloud provider outage
Component
Cloud load balancer Ingress controller
Contributing factor
–
Detection
Users reported slow performance of their applications in the #ask-cloud-platform Slack channel.
Remediation
AWS restarted the NLB health checking subsystem, which refreshed its list of targets and recovered the NLB in the affected Availability Zone
Action items
Raised mitigation tickets after a post-incident review, filed as post-aws-incident issues in the ministryofjustice/cloud-platform repository
View case
IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
OpenAI2022
Symptoms
At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.
Service outage High error rate Intermittent timeouts
Trigger
A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.
Root cause
Operator error
Component
Ingress controller Model hub
Contributing factor
No safeguard on destructive change No headroom Deferred upgrades Overlapping failures
Remediation
re-allocated the models from the lost cluster to the remaining two clusters
rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
built workarounds with Azure support after losing the connection to Azure Private Link
tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT
Action items
Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
Change how model weights are stored so they are more efficient and faster to load
Lessons
Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.
View case
IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
GitLab2022
Symptoms
At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.
High error rate High latency Intermittent timeouts
Trigger
A k8s-workloads/gitlab-com pipeline deployed merge request !1643, a GitLab chart bump that changed the port names of Kubernetes Services and Deployments (related to the Prometheus service and pod monitors), starting with the canary environment gprd-cny at 07:49 UTC.
Root cause
Configuration change
Component
Helm Cloud load balancer HAProxy
Contributing factor
Automation amplification
Detection
A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.
Remediation
drained the canary environment and re-enabled it after twenty minutes
cancelled the deployment jobs for clusters c and d before they started
drained gprd-us-east1-d in HAProxy and re-enabled it five minutes later
raised the minimum replica count above 100 (merge request !1650)
drained gprd-us-east1-c in HAProxy, deployed the change to it, and re-enabled it
reverted the minimum replica increase (merge request !1652)
Action items
Implement graceful draining of a whole regional cluster
Create a Chatops command that puts PagerDuty into maintenance mode
Lessons
Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes. GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.
View case
IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
Chatwork2021
Symptoms
In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.
Intermittent connection failures DNS resolution failures Intermittent timeouts Load imbalance
Trigger
During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.
Root cause
Capacity exhaustion
Component
conntrack table CoreDNS kube-proxy Client-side DNS resolver and cache VPC DNS resolver
Contributing factor
Unreviewed defaults Unrepresentative staging Unknown limit
Remediation
Raised the conntrack maxPerCore value in the kube-proxy configuration
Installed dns-autoscaler so CoreDNS ran with more pods
Installed node-local-dns as a DaemonSet on every node
Replaced force_tcp with prefer_udp in the node-local-dns configuration
Lessons
On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.
View case
IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNI
Microsoft Azure2020
Symptoms
In newly created AKS clusters using Azure CNI in Central India, kubectl exec and kubectl logs could time out, and pod-to-pod or pod-to-node traffic could time out. The cluster diagnostics page showed tunnel exec issues. Clusters using kubenet were not affected.
Intermittent timeouts
Trigger
VMs in Central India started being assigned MAC addresses in a much higher range than before (06 prefix rather than the usual 00).
Root cause
Software defect
Component
Azure CNI Node operating system image
Contributing factor
–
Remediation
told owners of affected scale-set clusters to run a node image upgrade, or any Kubernetes upgrade such as a patch version, to get the Azure CNI that does not use a bridge
told owners of availability-set clusters to scale out and delete the VMs with a high-range MAC address in bridge mode
advised recreating affected clusters that were brand new
Action items
Roll out a hotfix that makes transparent mode the default in all regions
Lessons
A change in how the platform assigns MAC addresses can wake a long-dormant bug in the network plugin. Knowing which CNI mode the nodes run (bridge or transparent) and keeping node images current decides whether such a bug reaches you.
View case

Check your cluster for this

These 26 cases involve Linux kernel and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap