Incident Library
Applied filters Root causeConfiguration changeFactorMonitoring blind spotDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases13 cases

1 to 5 of 13

  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026sev1
    Symptoms

    At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

    Service outageTLS certificate errors
    Trigger

    The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

    Root cause
    Configuration change
    Component
    cert-managerCDN and edge networkAuthoritative DNS servers and records
    Contributing factor
    No configuration testsMonitoring blind spotChange with a delayed effect
    Detection

    Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

    Remediation
    • Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate
    Action items
    • Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
    • Deploy the Pulumi update for the xPRO application in Production
    • Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
    • Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
    • Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires
    Lessons

    Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

    View case
  2. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
    Symptoms

    From 01:36 UTC on 12 March 2026, GitHub.com returned elevated error rates for Git operations, web requests and related services. At 01:54 UTC the GitHub status page said it was investigating reports of impacted performance for some services.

    High error rate
    Trigger

    During a planned infrastructure upgrade, a configuration issue made newly provisioned Kubernetes nodes run an incompatible version of etcd.

    Root cause
    Configuration change
    Component
    etcd
    Contributing factor
    Monitoring blind spot
    Detection

    The status page opened the incident at 01:54 UTC, 18 minutes after errors began, citing reports of impacted performance.

    Remediation
    • paused deployments for the duration of the incident
    • identified the newly provisioned nodes running the incompatible etcd version and removed them, which restored cluster consensus
    • ran a validation deploy that confirmed all systems were healthy before resuming normal operations
    Action items
    • Enforce version compatibility programmatically during node replacements
    • Add monitoring that detects split-brain conditions earlier
    • Update the recovery tooling to reduce restoration time
    View case
  3. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
    Symptoms

    Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

    Service outageHigh error rateControl plane unresponsiveDNS resolution failures
    Trigger

    At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

    Root cause
    Configuration change
    Component
    Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent
    Contributing factor
    Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access
    Detection

    OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

    Remediation
    • Scaled clusters down to reduce the total Kubernetes API load
    • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
    • Scaled up the Kubernetes API servers
    • Removed the telemetry service once enough control was back
    • Moved traffic from impacted clusters to healthy ones
    • Intervened manually in clusters where simultaneous resource downloads saturated resource limits
    Action items
    • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
    • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
    • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
    • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
    • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster
    Lessons

    Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

    View case
  4. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
    Symptoms

    Aqua's research team Nautilus recorded the attack on one of its Kubernetes honeypots. What appeared in the cluster was a DaemonSet running the image kuberntesio/kube-controller:1.0.1 from Docker Hub on every node, using the nodes' CPU to mine Monero.

    Unauthorised access or a foreign workloadResources deleted
    Trigger

    An API server misconfigured to accept unauthenticated requests from anonymous users with privileges.

    Root cause
    Configuration change
    Component
    Kubernetes API serverRBAC and service accountsContainer registry and its credentials
    Contributing factor
    Monitoring blind spot
    Detection

    Aqua recorded the attacker's API requests on its honeypot; planted AWS access keys raised a beacon when the attacker tried to use them.

    Lessons

    Fixing an exposed API server is not enough once an attacker has been in: a ClusterRoleBinding to an attacker-created ServiceAccount keeps working after anonymous access is closed. Attackers name these objects and images after real Kubernetes components so they blend into the API audit logs and pass for legitimate deployments. Aqua recommends scanning clusters for vulnerabilities, exposed secrets and misconfigurations with a tool such as Trivy.

    View case
  5. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
    Symptoms

    At 4:53 pm Pacific on 25 January 2023 one of OpenAI's GPU clusters had a major outage, and the text-davinci-003 model went fully down.

    Service outageThroughput degradation
    Trigger

    A configuration change to the CNI (Container Network Interface) plugins, which provide network connectivity to containers, was deployed to a GPU cluster that contained a hardware variant the staging environment did not have.

    Root cause
    Configuration change
    Component
    CNI plugin
    Contributing factor
    Unrepresentative stagingMonitoring blind spot
    Detection

    Engineers immediately identified the problem as network connectivity, but it took nearly an hour to identify CNI as the cause.

    Remediation
    • put a fix for the CNI configuration change in place, which restored 80% of traffic immediately
    • deployed models back onto GPUs over about three more hours until capacity was fully stable
    Action items
    • Make staging environments contain every type of hardware configuration so they better represent production
    • Monitor and alert on the health of the CNI system directly
    • Fix the network and storage bottlenecks that limit how quickly many replicas of a model can be deployed
    Lessons

    A change that had been deployed to other clusters over 24 hours and deemed safe still broke a cluster with different hardware, so staging needs every hardware configuration that production has.

    View case
Check your cluster for this

These 13 cases involve Kubernetes API server and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change13 cases

Component
Kubernetes API server4Cloud load balancer3Admission webhook2Client-side DNS resolver and cache2kubelet2Argo CD1CDN and edge network1cert-manager1Cilium1Cluster dashboard1CNI plugin1Configuration templating1conntrack table1CoreDNS1Authoritative DNS servers and records1etcd1external-dns1GKE control plane1HTTP and RPC client library1Ingress controller1Node launch template1Linux kernel1Container memory limit1Persistent volumes and stateful sets1Prometheus1RBAC and service accounts1Container registry and its credentials1Telemetry agent1Terraform1
Contributing factor
Monitoring blind spot13Excessive blast radius3Unrepresentative staging3Automation amplification2Unreviewed defaults2Change with a delayed effect2No configuration tests2Missed in review2Signal buried in noise1Unverified assumption1Ignored early signal1Cache-masked failure1Unknown limit1Limited debug access1No alert on limit1No health probes1No resource requests or limits1No runbook1No safeguard on destructive change1No staged rollout1Retry storm1
Organisation
DFDS3OpenAI2Aqua Security1Datadog1GitHub1InfluxData1Jetstack1JW Player1MIT Open Learning1NU.nl1
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
  3. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  4. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
  5. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  6. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  7. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
  8. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  9. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  10. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
  11. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  12. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard
    JW Player2019
  13. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019