Incident Library
Applied filters FactorSingle point of failureDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases33 cases

16 to 20 of 33

  1. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
    Symptoms

    About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

    Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag
    Trigger

    An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

    Root cause
    Breaking change in an upgrade
    Component
    CalicoKubernetes API serverAdmission webhooketcd
    Contributing factor
    Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure
    Detection

    The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

    Remediation
    • deleted the Open Policy Agent webhook configurations to stop API server timeouts
    • terminated all worker nodes of the cluster
    • restored the control plane from the pre-upgrade backup onto the first control plane node
    • switched off the cluster autoscaler until the control plane was stable
    • generated new TLS certificates so replacement control plane nodes could join etcd
    • brought traffic back in steps from 1% to 100%
    • restarted legacy services that had been stopped by hand
    Action items
    • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
    • Standardise components across clusters and replace bespoke configurations
    • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
    • Add pre-production clusters that can be tested with confidence
    • Improve the backup restore process and make cluster upgrades safe
    Lessons

    Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

    View case
  2. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022
    Symptoms

    A user of the LIS JupyterHub reported a "backoff after failed scale-up" error, and user servers would not start. An engineer opened incident #36, "LIS hub cannot scale", at 5:40 AM Pacific on 14 November 2022.

    Scale-up failurePods PendingService outage
    Trigger

    At 10 AM UK time on 14 November 2022, the B zone of Google Cloud's europe-west2 region (London) ran out of resources to allocate. The LIS cluster's nodes were restricted to that zone.

    Root cause
    Cloud provider outage
    Component
    Node poolCluster autoscaler
    Contributing factor
    Single point of failurePages not delivered
    Detection

    A user reported the "backoff after failed scale-up" error; an engineer opened the incident at 5:40 AM Pacific, about three hours and forty minutes after the zone ran out of resources.

    Remediation
    • edited the node pool in the GCP console to allow node creation in the A and C zones of europe-west2, which were not exhausted
    Action items
    • Either restrict the node pool back to the B zone after the Black Friday event, or move the cluster and the NFS Filestore to the A zone (infrastructure issue #1944)
    • Investigate an enterprise plan that allows a regional NFS, at higher cost (infrastructure issue #1945)
    • Make sure PagerDuty notifications always get through to engineers (team-compass issue #574)
    Lessons

    A node pool pinned to one zone for storage performance cannot scale when the cloud provider runs out of capacity in that zone. The autoscaler's "backoff after failed scale-up" message usually points to quotas, and raising quotas does nothing here; the ZONE_RESOURCE_POOL_EXHAUSTED error in the logs is what identifies a zone without capacity. The review found that one engineer wearing many hats (incident commander, comms and debugging), with pages not reaching the other responders, made the response harder; a team response would have given more support.

    View case
  3. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
    Symptoms

    At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.

    Service outageControl plane lagControl plane unresponsiveStale or inconsistent API readsCredentials unavailable
    Root cause
    Certificate or credential expiry
    Component
    GKE control planeKubernetes API serverNode poolRBAC and service accounts
    Contributing factor
    Limited debug accessSingle point of failure
    Detection

    The health check system sent a notification that the service was down; the team then found the main dashboard offline.

    Remediation
    • recreated the node pools by hand after the credential rotation failed to recreate them
    • duplicated the main cluster's services into a newly created GKE cluster
    • deleted the Kubernetes Service records in the old cluster so its load balancers could be attached to the new cluster
    • detached the StatefulSet from the old cluster and switched all writes to the main disk
    • exported the data from the temporary ClickHouse instance and inserted it into the main instance
    Action items
    • Change the deployment manifests so services can be switched between clusters quickly without a complicated persistent disk migration
    Lessons

    Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call. On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.

    View case
  4. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
    Symptoms

    At around 5.44pm UTC on 25 January 2022, Gitpod saw a high error rate when pulling images from Google Container Registry (gcr.io), with the error "cannot get manifest". About 10 minutes later users reported that they could not start workspaces.

    Image pull failuresDNS resolution failuresService outageData loss
    Root cause
    DNS or network fault
    Component
    CoreDNSContainer registry and its credentialsObject storage
    Contributing factor
    Single point of failure
    Detection

    The team noticed a high error rate on image pulls from GCR at around 5.44pm UTC; about 10 minutes later users reported that workspaces would not start, and the on-call team started incident response.

    Remediation
    • Created new workspace clusters in the EU and the US with a werft job
    • Redirected traffic to the new clusters
    Action items
    • Keep workspace data on cloud provider persistent volumes that survive node and cluster failures
    • Let users download their files from a workspace that cannot be started
    • Improve the workspace backup process
    • Use three DNS servers instead of one: a GCP-internal one, 8.8.8.8 and Cloudflare's 1.1.1.1
    • Do more DNS resiliency work, tracked in an epic
    • Improve support for multi-region clusters and later go multi-cloud
    View case
  5. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020
    Symptoms

    On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

    Service outageContainer crash loopTLS certificate errorsCredentials unavailable
    Trigger

    At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

    Root cause
    Certificate or credential expiry
    Component
    cert-managerSecrets store
    Contributing factor
    Single point of failureMonitoring blind spotUnverified assumptionHarmful manual action
    Detection

    Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

    Remediation
    • Forced cert-manager to recreate the Vault certificate
    • Restarted the Vault cluster by hand so it loaded the new certificate
    • Deleted some pods of critical workloads to skip their 5-minute exponential backoff
    Action items
    • Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
    • Deploy the Reloader controller to roll Vault when its certificate data changes
    • Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup
    Lessons

    A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

    View case
Check your cluster for this

These 33 cases involve Kubernetes API server and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause11

Dependency failure3 cases

Component
Cloud block storage1Container runtime1Credential broker for pods1Database1Data center power and cooling1etcd1InfiniBand fabric1Persistent volumes and stateful sets1Service discovery1Telemetry agent1
Contributing factor
Single point of failure3Monitoring blind spot2Automation amplification1Excessive blast radius1Unpatched known bug1Harmful manual action1No alert on limit1No runbook1Overlapping failures1
Organisation
Magic Pages1Nebius1Target1
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026
  2. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025
  3. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019

Cloud provider outage3 cases

Component
AKS control plane1CDN and edge network1Cloud load balancer1Cluster autoscaler1Database1Ingress controller1Node pool1Web application firewall rules1
Contributing factor
Single point of failure3Excessive blast radius1Pages not delivered1
Organisation
2i2c1ANDPAD1Microsoft Azure1
  1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed
    ANDPAD2026
  2. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  3. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022

Capacity exhaustion13 cases

Component
kubelet4Database3Ingress controller3Pod scheduling and placement3Kubernetes API server2Container memory limit2Node pool2Container registry and its credentials2Admission webhook1In-memory cache1CI system1Cloud controller manager1conntrack table1Container runtime1CoreDNS1Credential broker for pods1Client-side DNS resolver and cache1Helm1Istio1kube-dns1Linux kernel1Managed Kubernetes control plane and provider API1NAT gateway1Node root disk1Persistent volumes and stateful sets1
Contributing factor
Single point of failure13Unreviewed defaults4Monitoring blind spot3Retry storm3Automation amplification2Excessive blast radius2Unknown limit2Harmful manual action2No alert on limit2No caching2Unverified assumption1Ignored early signal1Limited debug access1No runbook1No headroom1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
Backpackr2Zalando2Datadog1IONOS Cloud1Magic Pages1Mercari1Moonlight1PostHog1Saltside1Scaleway1UK Ministry of Justice1
  1. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout
    IONOS Cloud2026
  2. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  3. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  4. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  5. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  6. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  7. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  8. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  9. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  10. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  11. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  12. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018
  13. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Configuration change1 case

Component
In-memory cacheContainer memory limitPersistent volumes and stateful sets
Contributing factor
Unverified assumptionNo health probesNo headroomRetry stormSingle point of failure
Organisation
AdGuard VPN
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025

Breaking change in an upgrade2 cases

Component
Admission webhook1Calico1ConfigMaps and Secrets1Client-side DNS resolver and cache1etcd1GKE control plane1Kubernetes API server1Node pool1Persistent volumes and stateful sets1
Contributing factor
Single point of failure2Untested backup1Configuration not in code1Monitoring blind spot1No headroom1Ineffective rollback1Deferred upgrades1Unrepresentative staging1
Organisation
Reddit1Wealthy1
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  2. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023

Software defect3 cases

Component
Kubernetes API server2Database1etcd1GPU driver1kubelet1Linkerd1Linux kernel1Container memory limit1Persistent volumes and stateful sets1
Contributing factor
Single point of failure3Monitoring blind spot2Unverified assumption1Change with a delayed effect1Ignored early signal1Harmful manual action1No staged rollout1Overlapping failures1Ineffective rollback1
Organisation
Barush Mendez1Monzo1Palark1
  1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  2. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  3. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Hardware failure1 case

Component
ConfigMaps and SecretsetcdKubesprayNode root disk
Contributing factor
Single point of failure
Organisation
Hans H. Kim
  1. IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joined
    Hans H. Kim2023

Certificate or credential expiry2 cases

Component
cert-manager1GKE control plane1Kubernetes API server1Node pool1RBAC and service accounts1Secrets store1
Contributing factor
Single point of failure2Unverified assumption1Limited debug access1Harmful manual action1Monitoring blind spot1
Organisation
GoCardless1Webhook Relay1
  1. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
  2. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020

DNS or network fault2 cases

Component
Container registry and its credentials2CoreDNS1Container memory limit1Object storage1
Contributing factor
Single point of failure2Harmful manual action1Overlapping failures1
Organisation
GitHub1Gitpod1
  1. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
  2. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020

Faulty deployment2 cases

Component
Admission webhook1CI system1Ingress controller1Node pool1
Contributing factor
Single point of failure2Unverified assumption1Change started before off-hours1Change with a delayed effect1No alert on limit1
Organisation
Kubernetes project1UK Ministry of Justice1
  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
  2. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018