Incident Library
Applied filters SymptomHigh error rateDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases62 cases

25 to 29 of 62

  1. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic
    HubSpot2024
    Symptoms

    Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

    Probe failuresHigh latencyHigh error rate
    Trigger

    Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

    Root cause
    Software defect
    Component
    Ingress controllerLiveness and readiness probes
    Contributing factor
    Automation amplificationNo headroom
    Detection

    Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

    Remediation
    • Tried to increase the capacity of the remaining load balancer instances
    • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal
    Action items
    • Change the health check configuration so health checks do not time out falsely
    • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
    • Increase the capacity of the load balancers for sudden traffic spikes
    • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
    • Update the status page automatically for large-scale outages
    • Integrate the monitoring systems more tightly with the communication tools
    • Expand load testing to simulate high-traffic scenarios
    • Introduce chaos engineering to test platform resilience regularly
    • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
    • Add auto-scaling that adjusts resource capacity to traffic patterns
    Lessons

    HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

    View case
  2. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023sev1
    Symptoms

    Standard alerting paged Honeycomb's on-call engineers about the performance and stability of event ingestion, and the automated load-shedding mechanisms tripped and sent further notifications. The status page described a higher than normal error rate and response time on the ingestion service.

    High error rateHigh latencyContainer crash loopService outageDatabase overload
    Trigger

    An abnormally large number of queries arrived from many separate connections within a few minutes.

    Root cause
    Capacity exhaustion
    Component
    DatabaseIn-memory cache
    Contributing factor
    Retry stormAutomation amplification
    Detection

    Standard alerting notified the on-call engineers about ingestion performance and stability, and the automated load-shedding mechanisms added notifications when they tripped. The first status post, about alerts on a higher than normal error rate and response time, went up at 18:28 PST (02:28 UTC), 13 minutes after the 02:15 UTC start.

    Remediation
    • Scaled the ingest fleet up manually and aggressively
    • Cut off most ingest for a few minutes so the ingest hosts could come back with enough capacity
    • Temporarily lowered the rate limits for the most aggressive traffic sources until their traffic subsided
    Action items
    • Keep the ingest fleet scaled up to prevent a repeat while longer-term fixes are worked on
    • Decide on the projects that will address these failure paths
    Lessons

    Load shedding and auto-scaling were in place, yet a burst that arrived faster than the cache could spread its writes still took the ingest hosts down. During recovery, Kubernetes' crash loop back-off worked against the operators: the hosts that had failed stayed offline while the fleet needed every host. Clients that retry whole OTel batches after a partial drop can multiply traffic many times over, so per-source rate limits were what finally stabilised ingest.

    View case
  3. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
    Symptoms

    From about 10:30 UTC on 30 August 2023, customers had trouble accessing or using Azure, Microsoft 365 and Power Platform services in Australia East. The affected Azure services included Azure Kubernetes Service (AKS) along with storage, SQL Database, Cosmos DB, virtual machines and Azure Resource Manager.

    Control plane unresponsiveHigh error rateService outage
    Trigger

    At 08:41 UTC a lightning strike on electrical infrastructure about 18 miles away caused a utility voltage sag that shut down the cooling chillers of datacenters in one Australia East availability zone; 13 chillers failed to restart automatically and five could not be restarted, so two data halls overheated and their infrastructure was shut down.

    Root cause
    Cloud provider outage
    Component
    AKS control planeDatabase
    Contributing factor
    Single point of failureExcessive blast radius
    Detection

    Cosmos DB impact was detected by monitoring at 10:47 UTC, first automated communications went to Azure Service Health at 10:48 UTC, the public Azure Status page was updated at 11:30 UTC, and all subscriptions in Australia East were sent portal communications at 11:36 UTC.

    Remediation
    • shut down the remaining networking, compute and storage in the two affected data halls so the chilled water loop could cool down
    • restarted the five chillers manually at 12:12 UTC
    • restored power to all hardware by 15:10 UTC and brought infrastructure back in phases
    • restored AKS operations through the recovery of the SQL Database operation queue, with no other mitigation
    Action items
    • AKS: convert all operation queue SQL Database databases to be zone redundant (estimated September 2023)
    • AKS: replace all cross-region SQL Database queue usage with zone-redundant Service Bus queues (estimated September 2023)
    • Change the control timing logic of the Chiller Management System so chillers do not fail to restart after voltage fluctuations
    • Change the emergency procedure for manual restarts of simultaneous chiller failures from an adjacent sequence to a load-based sequence per data hall
    • Increase technician staffing at the datacenter as a temporary measure for manual chiller restarts
    • Storage: optimise the large-scale recovery process and update troubleshooting guides (estimated December 2023)
    • Azure Service Fabric: improve reliability of SQL Database tenant ring recovery (estimated December 2023)
    • SQL Database: review the trigger criteria of auto-failover groups (estimated October 2023)
    • SQL Database: upgrade internal tooling to enable mass migration of databases (estimated December 2023)
    • Cosmos DB: optimise Service Managed Failover for single-region-write accounts (estimated November 2023)
    • ARM: complete the migration of its storage layer to the zonally redundant architecture (estimated December 2023)
    • Incident management: harden readiness, process and playbook for power-down scenarios (estimated October 2023)
    Lessons

    A control plane spread across availability zones still fails with a zone when one of its dependencies, here the SQL Database operation queue, is not zone redundant. When a second region uses the same database, the failure crosses into that region as well. Workloads already running were not expected to be affected; what stopped was everything that needs the managed service's control plane, such as scaling and cluster changes.

    View case
  4. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023
    Symptoms

    When MonotaRO raised the share of requests for the remaining endpoints of its backend API sent to EKS from 1% to 25%, Sentry filled with errors saying the API could not reach the database and outside services. The errors were about DNS name resolution, and the traffic had to be shifted back to the EC2 version.

    DNS resolution failuresHigh error rate
    Trigger

    MonotaRO raised the share of requests for the remaining API endpoints routed by the ALB to the EKS version from 1% to 25%.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSClient-side DNS resolver and cacheContainer CPU limitHorizontal pod autoscalerPod termination
    Contributing factor
    Unreviewed defaultsNo resource requests or limitsUnrepresentative stagingUnverified assumption
    Detection

    Sentry showed a large number of errors about connecting to the database and outside services right after the traffic share was raised.

    Remediation
    • Shifted the requests back to the EC2 version to stop the errors
    • Added an HPA to autoscale the CoreDNS pods as a stopgap, which let the migration to EKS finish
    Action items
    • Set ndots to 1 through dnsConfig in the application Deployments
    • Set cache 600 in the Corefile in the coredns ConfigMap to lengthen the CoreDNS cache TTL
    • Add lameduck 30s to the CoreDNS health plugin so CoreDNS keeps serving while it is scaled in
    • Consider reducing the differences between production and staging, weighed against cost
    Lessons

    MonotaRO assumed that a managed service would not cause problems and so skipped researching other companies' migrations, load testing and monitoring. Staging differed from production in its network layout, so ndots differed between them, and staging databases could not take a high-load test. Looking at the CoreDNS logs and resolv.conf instead of guessing found that ndots:5 made lookups of outside names try every search domain first.

    View case
  5. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023
    Symptoms

    On 9 August 2023, right after a MediaWiki configuration deployment at 11:41 UTC, the MediaWiki deployments on Kubernetes terminated TLS with a wrong certificate, and about 1% of Wikimedia web and API traffic, around 120 requests per second, received errors for about 30 minutes. The issue was detected by a page for failed service probes on mw-api-ext and mw-api-int.

    High error rateTLS certificate errorsControl plane lag
    Trigger

    At 09:37 a change to the global Kubernetes defaults was merged that switched supported charts to cert-manager certificates. The next MediaWiki on Kubernetes deployment, a configuration change at 11:41, picked it up.

    Root cause
    Configuration change
    Component
    cert-managerHelm
    Contributing factor
    Change with a delayed effectNo configuration tests
    Detection

    A page from the ProbeDown alerts for mw-api-ext:4447 and mw-api-int:4446 at 11:47; a KubernetesAPILatency alert had fired at 11:46.

    Remediation
    • Identified the issue and prepared a patch, https://gerrit.wikimedia.org/r/947331
    • Redeployed the MediaWiki deployments on Kubernetes
    Action items
    • Review whether deployment procedures and testing should be strengthened, for example against surprising changes on the next deployment and with canary deployments for Kubernetes
    • Look at metrics that become unavailable or unhealthy during deployments, or at reducing the impact of deployments
    Lessons

    The cause was found quickly because the person who made the breaking change was responding. The change should have been caught by CI, but the repository it was made in (hieradata) does not run the CI of the deployment charts. Only 1% of traffic going to Kubernetes kept the impact small, and the low error volume made it slower to link the errors to Kubernetes.

    View case
Check your cluster for this

These 62 cases involve Database and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause10

Configuration change14 cases

Component
Cloud load balancer2Client-side DNS resolver and cache2Helm2Kubernetes API server2Pod priority and preemption2Telemetry agent2Argo CD1cert-manager1Cilium1Configuration templating1ConfigMaps and Secrets1conntrack table1Container runtime1CoreDNS1Database1etcd1external-dns1HAProxy1HTTP and RPC client library1Ingress controller1kubelet1Linux kernel1Application logging library1Container memory limit1Persistent volumes and stateful sets1Prometheus1Terraform1AWS VPC CNI1
Contributing factor
Monitoring blind spot5Unverified assumption4Excessive blast radius4Automation amplification3Change with a delayed effect3Limited debug access2No configuration tests2No staged rollout2Retry storm2Missed in review2Unreviewed defaults1Cache-masked failure1Unknown limit1No alert on limit1No resource requests or limits1No runbook1No safeguard on destructive change1No headroom1Partial rollout1Unrepresentative staging1
Organisation
OpenAI3Datadog1DFDS1GitHub1GitLab1Grafana Labs1InfluxData1LIFULL1MUSINSA1NU.nl1Pusher1Wikimedia Foundation1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
  3. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
  4. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  5. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  6. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023
  7. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  8. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
  9. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  10. IL-0442 Older pods on LIFULL's KEEL Kubernetes cluster were preempted all at once and the external error rate spiked
    LIFULL2021
  11. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  12. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  13. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  14. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Capacity exhaustion21 cases

Component
Database6CoreDNS4Kubernetes API server4Node pool4Container memory limit3Pod scheduling and placement3Cloud block storage2In-memory cache2CI system2Client-side DNS resolver and cache2etcd2Helm2Jobs and CronJobs2Persistent volumes and stateful sets2Calico1conntrack table1Container CPU limit1Credential broker for pods1GKE control plane1HAProxy1Horizontal pod autoscaler1Model inference server1Ingress controller1JupyterHub1kube-proxy1kubelet1Linux kernel1Managed Kubernetes control plane and provider API1Pod termination1Container registry and its credentials1Service discovery1Telemetry agent1
Contributing factor
Unreviewed defaults5Single point of failure5Unverified assumption4Unknown limit4Monitoring blind spot4No alert on limit4Retry storm4Excessive blast radius3Unrepresentative staging3Automation amplification2Change during peak hours2Change with a delayed effect2Harmful manual action2No capacity planning2No headroom2No caching1No resource requests or limits1No runbook1No staged rollout1Ineffective rollback1
Organisation
Zalando32i2c1Civis Analytics1CyberAgent1freedesktop.org1GitHub1GitLab1Google Cloud1Honeycomb1IONOS Cloud1loveholidays1Microsoft1MonotaRO1Moonlight1PostHog1RebelMouse1Render1UC Berkeley1Wikimedia Foundation1
  1. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
    GitHub2026
  2. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026
  3. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout
    IONOS Cloud2026
  4. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  5. IL-0413 LLM inference engines timed out or ran out of resources under load in about 60% of 156 severe incidents at Microsoft
    Microsoft2025
  6. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  7. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  8. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024
  9. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  10. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023
  11. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
  12. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  13. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  14. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  15. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  16. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  17. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  18. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  19. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018
  20. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage
    freedesktop.org2018
  21. IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchanged
    UC Berkeley2017

Cloud provider outage5 cases

Component
Database3Cloud load balancer2AKS control plane1CDN and edge network1Cluster autoscaler1GKE control plane1Horizontal pod autoscaler1Ingress controller1Node root disk1Persistent volumes and stateful sets1Web application firewall rules1
Contributing factor
Overlapping failures2Single point of failure2Unverified assumption1Excessive blast radius1Unpatched known bug1Harmful manual action1No headroom1Health check tied to a dependency1Retry storm1
Organisation
ANDPAD1Grafana Labs1Microsoft Azure1OpenAI1Twingate1
  1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed
    ANDPAD2026
  2. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster
    Twingate2025
  3. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025
  4. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  5. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019

Breaking change in an upgrade4 cases

Component
Admission webhook1Calico1etcd1Model inference server1Ingress controller1Istio1Kubernetes API server1Linux kernel1Web application firewall rules1
Contributing factor
Unrepresentative staging2Untested backup1Configuration not in code1Ignored early signal1Monitoring blind spot1Single point of failure1
Organisation
GitHub1Reddit1UK Ministry of Justice1Wikimedia Foundation1
  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  2. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026
  3. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
  4. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023

Faulty deployment2 cases

Component
In-memory cache1CI system1Cloud load balancer1ConfigMaps and Secrets1Database1Helm1Ingress controller1Pod termination1
Contributing factor
Unverified assumption2Signal buried in noise1Unreviewed defaults1Monitoring blind spot1Dependence on a warm cache1
Organisation
Wikimedia Foundation1Yonghyun Kim1
  1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  2. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster
    Wikimedia Foundation2025

Data loss or corruption1 case

Component
Container registry and its credentials
Contributing factor
Cache-masked failure
Organisation
Harness
  1. IL-0474 Harness Feature Flags auth returned 502s when rescheduled pods could not get an image deleted from Artifact Registry
    Harness2025

Software defect9 cases

Component
kubelet3Cloud load balancer2etcd2Ingress controller2Admission webhook1Container CPU limit1Database1Authoritative DNS servers and records1Liveness and readiness probes1Horizontal pod autoscaler1Kubernetes API server1Linkerd1Linux kernel1Node pool1Serverless function platform1
Contributing factor
Monitoring blind spot3Automation amplification2Change with a delayed effect2Unpatched known bug2Unknown limit2Excessive blast radius1Unreviewed defaults1Ignored early signal1Harmful manual action1No configuration tests1No runbook1No safeguard on destructive change1No headroom1Overlapping failures1Retry storm1Ineffective rollback1Single point of failure1
Organisation
Amazon Web Services2Airbnb1AirMap1DFDS1Grafana Labs1HubSpot1Monzo1Omio1
  1. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  2. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic
    HubSpot2024
  3. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning
    Amazon Web Services2023
  4. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  5. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
  6. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020
  7. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  8. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  9. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Certificate or credential expiry1 case

Component
Cluster autoscalerManaged Kubernetes control plane and provider API
Contributing factor
Change with a delayed effect
Organisation
Linode
  1. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025

DNS or network fault3 cases

Component
conntrack table2CoreDNS2kube-proxy2Cloud load balancer1Client-side DNS resolver and cache1Horizontal pod autoscaler1Ingress controller1Pod termination1Telemetry agent1
Contributing factor
Unreviewed defaults2Ignored early signal2Unverified assumption1Automation amplification1Unknown limit1No caching1
Organisation
Adevinta1Lotte E-commerce1Preply1
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  2. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
  3. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020

Operator error2 cases

Component
Database1Ingress controller1Model hub1
Contributing factor
No safeguard on destructive change2No headroom1Overlapping failures1Deferred upgrades1
Organisation
GitLab1OpenAI1
  1. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  2. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022