Incident Library
Applied filters SymptomService outageFactorNo headroomClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

9 to 9 of 9

  1. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
    Symptoms

    On Friday 19 July 2019 the Grafana Cloud Hosted Prometheus service, which is built on the Cortex project, was down for 26 minutes. New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start.

    Pod evictionHigh error rateService outageOOMKilled
    Trigger

    An engineer created a new dedicated Cortex cluster for a large customer. Its configuration did not include the new pod priorities, so all its pods got the default medium priority, and the Kubernetes cluster did not have enough free resources to fit it.

    Root cause
    Configuration change
    Component
    Pod priority and preemption
    Contributing factor
    Partial rolloutAutomation amplificationMissed in reviewNo headroomRetry storm
    Detection

    New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start of the outage.

    Remediation
    • Scaled up the underlying Kubernetes cluster to fit both the new and the production Cortex clusters
    • Diagnosed and fixed the out-of-memory errors in the authenticating reverse proxies in front of Cortex
    Action items
    • Give the production ingesters the high priority class, which should have been done before medium became the default
    • Have more people review changes to configuration that is global to the cluster
    • Require a design document for every configuration change outside a namespace
    • Automate the sizing of the frontend authenticating reverse proxy so it does not run out of memory under overload
    • Investigate the default backoff and scaling behaviour of Prometheus to prevent a stampeding herd of retries
    Lessons

    Medium should not have become the default priority until the production ingesters had the high priority, and that designation should have reached production sooner. The change was considered too minor for a design document and was only discussed in a GitHub issue; cluster-global configuration needs more scrutiny. On the positive side, once Cortex had enough capacity it recovered by itself, and Grafana Loki helped confirm that the ingesters exited gracefully.

    View case
Check your cluster for this

These 9 cases involve Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause7

Capacity exhaustion1 case

Component
CoreDNSHorizontal pod autoscalerContainer memory limitNode pool
Contributing factor
Automation amplificationMonitoring blind spotNo headroomRetry storm
Organisation
Buildkite
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026

Configuration change2 cases

Component
In-memory cache1Container memory limit1Persistent volumes and stateful sets1Pod priority and preemption1
Contributing factor
No headroom2Retry storm2Unverified assumption1Automation amplification1No health probes1Partial rollout1Missed in review1Single point of failure1
Organisation
AdGuard VPN1Grafana Labs1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019

Breaking change in an upgrade1 case

Component
ConfigMaps and SecretsClient-side DNS resolver and cacheGKE control planeNode poolPersistent volumes and stateful sets
Contributing factor
No headroomIneffective rollbackSingle point of failureDeferred upgrades
Organisation
Wealthy
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025

Faulty deployment1 case

Component
Cluster autoscalerDeschedulerNode pool
Contributing factor
Unreviewed defaultsNo alert on limitNo headroom
Organisation
UK Ministry of Justice
  1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025

Cloud provider outage1 case

Component
Kubernetes API server
Contributing factor
No capacity planningNo headroomOverlapping failures
Organisation
Namespace
  1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024

Software defect2 cases

Component
Cluster autoscaler1Application database1Container memory limit1Model hub1Model request routing1
Contributing factor
No headroom2No alert on limit1Overlapping failures1Retry storm1
Organisation
Hugging Face1OpenAI1
  1. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times
    Hugging Face2024
  2. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023

Operator error1 case

Component
Ingress controllerModel hub
Contributing factor
No safeguard on destructive changeNo headroomOverlapping failuresDeferred upgrades
Organisation
OpenAI
  1. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022