Incident Library
Applied filters SymptomHigh latencyFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases7 cases

6 to 7 of 7

  1. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
    Symptoms

    At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.

    High error rateHigh latencyIntermittent timeouts
    Trigger

    A k8s-workloads/gitlab-com pipeline deployed merge request !1643, a GitLab chart bump that changed the port names of Kubernetes Services and Deployments (related to the Prometheus service and pod monitors), starting with the canary environment gprd-cny at 07:49 UTC.

    Root cause
    Configuration change
    Component
    HelmCloud load balancerHAProxy
    Contributing factor
    Automation amplification
    Detection

    A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.

    Remediation
    • drained the canary environment and re-enabled it after twenty minutes
    • cancelled the deployment jobs for clusters c and d before they started
    • drained gprd-us-east1-d in HAProxy and re-enabled it five minutes later
    • raised the minimum replica count above 100 (merge request !1650)
    • drained gprd-us-east1-c in HAProxy, deployed the change to it, and re-enabled it
    • reverted the minimum replica increase (merge request !1652)
    Action items
    • Implement graceful draining of a whole regional cluster
    • Create a Chatops command that puts PagerDuty into maintenance mode
    Lessons

    Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes. GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.

    View case
  2. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019sev1
    Symptoms

    An upgrade of the network subsystem under Target's datacenter OpenStack, meant to cause only a brief disruption, ran into problems and left network connectivity disrupted for several hours. Kafka, which runs on that OpenStack, became intermittently available.

    Node NotReadyLoad imbalanceHigh latencyCredentials unavailable
    Trigger

    A network subsystem upgrade in the datacenter OpenStack infrastructure disrupted connectivity for several hours and made Kafka intermittently available.

    Root cause
    Dependency failure
    Component
    Container runtimeService discoveryTelemetry agentCredential broker for pods
    Contributing factor
    Excessive blast radiusAutomation amplificationSingle point of failureUnpatched known bug
    Remediation
    • enabled gossip encryption on Consul so messages from the poisoned mesh were rejected
    • stabilized the deployment engine and redeployed the development environment with encryption, terminating instances without it
    • upgraded Consul in place to 1.2.3 and then to 1.3.0
    Action items
    • Reduce how much of an outage a single Docker daemon can cause, together with running smaller and more numerous clusters
    Lessons

    Smaller and more numerous clusters limited the damage: the smaller development clusters and production were not hit the same way as the one large cluster, and dev and prod on one cluster would have been far worse. A Docker daemon shared by many workloads is a brittle failure point. Per-workload logging and metrics sidecars were judged the right design; a shared cluster-level agent would almost certainly have made the problem worse and harder to fix.

    View case
Check your cluster for this

These 7 cases involve Application database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Faulty deployment1 case

Component
In-memory cacheApplication database
Contributing factor
Automation amplificationChange during peak hoursMonitoring blind spotNo runbookHealth check tied to a dependencyRetry storm
Organisation
Microsoft Azure
  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026

Software defect2 cases

Component
Automated capacity manager1Cloud load balancer1Application database1Authoritative DNS servers and records1
Contributing factor
Automation amplification2Change with a delayed effect1Monitoring blind spot1No runbook1No safeguard on destructive change1Retry storm1
Organisation
Amazon Web Services1OpenAI1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025

Capacity exhaustion2 cases

Component
In-memory cache1CoreDNS1Application database1Horizontal pod autoscaler1Karpenter1Pod scheduling and placement1
Contributing factor
Automation amplification2Harmful manual action1Retry storm1
Organisation
Honeycomb1이언철1
  1. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  2. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023

Configuration change1 case

Component
Cloud load balancerHAProxyHelm
Contributing factor
Automation amplification
Organisation
GitLab
  1. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022

Dependency failure1 case

Component
Container runtimeCredential broker for podsService discoveryTelemetry agent
Contributing factor
Automation amplificationExcessive blast radiusUnpatched known bugSingle point of failure
Organisation
Target
  1. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019