Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases24 cases

6 to 10 of 24

  1. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025
    Symptoms

    On 1 April 2025 users of Linode Kubernetes Engine (LKE) began having trouble connecting to their clusters' control planes, and the LKE Dashboard was affected as well. Services inside the clusters kept working. The first status update was posted at 20:28 UTC; at 21:33 UTC Linode said it was investigating an issue with DNS functionality within LKE.

    DNS resolution failuresControl plane unresponsive
    Trigger

    An earlier update to a database that the LKE DNS system depends on caused the DNS service to hang.

    Root cause
    Dependency failure
    Component
    Authoritative DNS servers and recordsDatabase
    Contributing factor
    –
    Detection

    The first status update, at 20:28 UTC on 1 April, said Linode was investigating and that users might have difficulty connecting to their cluster control plane. At 21:33 UTC Linode said the issue affected DNS functionality within LKE, and at 22:29 UTC that it had identified the underlying cause.

    Remediation
    • restarted the affected LKE DNS services
    Action items
    • Improve how the DNS services recover from transient connectivity failures to their dependencies
    • Improve monitoring to detect similar problems sooner
    • Update the deployment process to better coordinate changes between dependent systems
    Lessons

    A change to a database that the DNS service depends on can make the DNS service hang, which here cut off external access to the clusters and the LKE Dashboard while services inside the clusters kept working. The DNS service should recover by itself from transient failures of its dependencies, and changes to dependent systems need to be coordinated.

    View case
  2. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster
    Wikimedia Foundation2025
    Symptoms

    On 2025-03-12 from 10:58 UTC, alerts for high backend response times came in, and the MediaWiki deployments mw-api-int, mw-parsoid and mw-jobrunner, which serve MediaWiki for non-user-facing purposes, showed high latency and database errors. The External Storage cluster, whose hosts store the compressed text of wiki page revisions, had a sharp rise in load and connections. The same happened again on 2025-03-13 from 11:15 UTC.

    Database overloadHigh latencyHigh error rateLog floodMonitoring outage
    Trigger

    At 10:44 UTC on 12 March a Scap deployment of the PHP 8.1 rollout for mw-api-int, mw-parsoid and mw-jobrunner was run with the flag -Dbuild_mw_container_image:False; the rollout was deployed with Scap again at 11:15 UTC on 13 March.

    Root cause
    Faulty deployment
    Component
    DatabaseIn-memory cacheConfigMaps and SecretsHelmCI system
    Contributing factor
    Unverified assumptionDependence on a warm cacheSignal buried in noise
    Detection

    Alerts for high backend response times started at 10:58 UTC on 12 March, 14 minutes after the Scap deployment; 2 on-call engineers were paged. The MediaWikiMemcachedHighErrorRate alert that pointed at the cause fired only briefly because its log pipeline was overloaded.

    Remediation
    • Deployed a patch to cirrus-streaming-updater that reduced SUP parallelism
    • Reduced the concurrency of the categoryMembershipChange job in changeprop-jobqueue
    • On 12 March, reverted 1126607 and 1126650 and re-ran Scap
    • On 13 March, rolling-restarted changeprop-jobqueue
    • Rolling-restarted mw-api-int
    • Disabled the categorymembership job in changeprop-jobqueue
    • Reverted 1127476 and 1127478
    • Redeployed mw-api-int to pick up the revert
    • Increased the number of mw-api-int replicas
    • Redeployed mw-jobrunner to pick up the revert
    • Redeployed mw-parsoid to pick up the revert
    Action items
    • Reduce the number of messages sent through channel:Memcached during failures
    • Add helm rollback functionality to Scap
    • Decide whether Scap should update helmfile-defaults when -Dbuild_mw_container_image:False is set
    • Consider removing envvars.inc from MediaWiki images
    • Consider "Why" dashboards that give an overview of a group of related components together with relevant links to follow
    • Coordinate incident responders better
    • Be able to work with MediaWiki developers during incidents
    Lessons

    The write-up puts the weight not on the technical root cause but on why the team could not see, even after the 12 March incident, that three MediaWiki deployments were running without Memcached. Alert dashboards showed what was happening, such as many busy PHP workers, but not why; dashboards linking components that affect each other, such as SQLBlobStore keys and External Storage, would have helped. The alert that pointed in the right direction did not stay firing and would probably have been lost among 78 others. The incident coordinator could not tell who was working on what or who had joined or left, the channels were noisy, the external impact was judged more critical than it was, and trace.wikimedia.org was not used although it might have helped.

    View case
  3. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025
    Symptoms

    From 3:30am to 7:08am PT, chatgpt.com and a small number of OpenAI APIs returned elevated error rates. The status page classed the incident as a full outage.

    High error rateContainer crash loopProbe failures
    Trigger

    A failure of Cosmos DB in ChatGPT, at OpenAI's service provider.

    Root cause
    Cloud provider outage
    Component
    Database
    Contributing factor
    Health check tied to a dependency
    Remediation
    • the service provider recovered the affected Cosmos DB databases at 4:25am PT
    • fixed the web server's health checking logic, which ended the pod crash loop
    Action items
    • Change the web systems to degrade more gracefully when they fail, including concurrency control and better load shedding
    • Make the database layer more resilient to service provider outages (described as already underway)
    Lessons

    The database came back at 4:25am PT, yet users saw errors until 7:08am PT: the web tier's own health checks kept removing pods after the dependency had recovered. The planned changes target the web tier's behaviour under failure, with concurrency control and load shedding, rather than the database alone.

    View case
  4. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025
    Symptoms

    On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

    Service outage
    Trigger

    One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

    Root cause
    Dependency failure
    Component
    DatabasePersistent volumes and stateful sets
    Contributing factor
    Harmful manual actionSingle point of failureMonitoring blind spot
    Remediation
    • Restarted the storage server to fix PVC provisioning
    • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
    • Cleaned up the corrupted cluster metadata
    • Redeployed MySQL Router
    Action items
    • Monitor the health of the storage system more closely
    • Improve the failover procedures for MySQL Router
    • Investigate and fix the status page so it shows reliable information
    Lessons

    The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

    View case
  5. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
    Symptoms

    On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

    Network unreachablePacket lossNode NotReady
    Trigger

    At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API
    Contributing factor
    Excessive blast radiusAutomation amplification
    Detection

    Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

    Remediation
    • Fixed both VPC route reflectors by manual action
    Action items
    • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
    • Run different software versions on the route reflectors so a single bug cannot stop several instances
    • Investigate the software issue that left the route reflectors in a stale state
    View case
Check your cluster for this

These 24 cases involve Database and Persistent volumes and stateful sets. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause7

Configuration change1 case

Component
Container runtimeDatabaseApplication logging libraryTelemetry agent
Contributing factor
Unverified assumptionExcessive blast radius
Organisation
MUSINSA
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026

Capacity exhaustion10 cases

Component
Database10In-memory cache3Container memory limit2Pod scheduling and placement2Admission webhook1BGP routing1CNI plugin1InfiniBand fabric1Jobs and CronJobs1JupyterHub1Linux kernel1Model hub1NAT gateway1Network control plane (SDN)1NIC, cable and switch port link1Node pool1Persistent volumes and stateful sets1Container registry and its credentials1Service discovery1Telemetry agent1
Contributing factor
Automation amplification4Excessive blast radius3Single point of failure3Unreviewed defaults2Harmful manual action2No alert on limit2Retry storm2Unverified assumption1Change with a delayed effect1Unknown limit1Monitoring blind spot1No resource requests or limits1No staged rollout1Overlapping failures1Ineffective rollback1Unrepresentative staging1
Organisation
2i2c1CyberAgent1Datadog1DFDS1GitLab1Honeycomb1incident.io1IONOS Cloud1Moonlight1PostHog1
  1. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026
  2. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
  3. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  5. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  6. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes
    incident.io2023
  7. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
  8. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  9. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  10. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019

Software defect4 cases

Component
Database4BGP routing1Cloud load balancer1Authoritative DNS servers and records1Linux kernel1Managed Kubernetes control plane and provider API1Container memory limit1Network control plane (SDN)1Network switch1Node operating system image1Persistent volumes and stateful sets1
Contributing factor
Automation amplification2Unverified assumption1Excessive blast radius1Unpatched known bug1No runbook1No staged rollout1Retry storm1Single point of failure1
Organisation
Amazon Web Services1Palark1Scaleway1Shopify1
  1. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
  3. IL-0462 Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacement
    Shopify2021
  4. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020

Dependency failure2 cases

Component
Database2Authoritative DNS servers and records1Persistent volumes and stateful sets1
Contributing factor
Harmful manual action1Monitoring blind spot1Single point of failure1
Organisation
Linode1Magic Pages1
  1. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025
  2. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025

Faulty deployment1 case

Component
In-memory cacheCI systemConfigMaps and SecretsDatabaseHelm
Contributing factor
Signal buried in noiseUnverified assumptionDependence on a warm cache
Organisation
Wikimedia Foundation
  1. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster
    Wikimedia Foundation2025

Cloud provider outage4 cases

Component
Database4AKS control plane1Helm1Node root disk1Persistent volumes and stateful sets1
Contributing factor
Unverified assumption2Excessive blast radius2Unpatched known bug1Overlapping failures1Health check tied to a dependency1Single point of failure1
Organisation
Devsisters1Grafana Labs1Microsoft Azure1OpenAI1
  1. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025
  2. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  3. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021
  4. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019

Operator error2 cases

Component
Database2JupyterHub1Persistent volumes and stateful sets1
Contributing factor
No safeguard on destructive change2Harmful manual action1Unsafe failure behaviour1
Organisation
GitLab1UC Berkeley1
  1. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  2. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017