Incident Library
Applied filters Root causeFaulty deploymentFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
    OpenAI2025
    Symptoms

    Between 2:10 PM and 3:05 PM PDT on July 9, 2025, success rates of the gpt-4.1-mini model in the API dropped for about 20 minutes, and ChatGPT search returned many more errors than usual for about 35 minutes.

    High error rate
    Trigger

    A faulty code change was deployed to a control plane cluster whose main job is managing low-priority workloads.

    Root cause
    Faulty deployment
    Component
    –
    Contributing factor
    Excessive blast radiusNo safeguard on destructive change
    Remediation
    • rolled back the faulty code change
    Action items
    • Add safeguards that prevent production inference workloads from being managed by the low-priority control plane cluster
    • Strengthen testing for deploying high-impact control plane changes
    • Move all existing production inference workloads under the production control plane cluster
    • Build a circuit breaker that stops large-scale unintended changes
    View case
  2. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
    Symptoms

    At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.

    Service outageResources deleted
    Trigger

    An update deployed to a test Kubernetes cluster also removed Traefik and Calico resources from CircleCI's primary production cluster.

    Root cause
    Faulty deployment
    Component
    Ingress controllerCalico
    Contributing factor
    Excessive blast radiusNo runbookMonitoring blind spot
    Detection

    CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.

    Remediation
    • restored Traefik in the production cluster
    • restored Calico in the production cluster, after which pods served customer requests again
    Action items
    • Improve network-level documentation and runbooks to cover every layer of the Kubernetes networks
    • Improve monitoring and alerting for the network infrastructure
    • Add guardrails that isolate the production and staging clusters so they cannot interfere with each other
    Lessons

    A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again. Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.

    View case
  3. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
    Symptoms

    An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

    Probe failuresService outage
    Trigger

    A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

    Root cause
    Faulty deployment
    Component
    Telemetry agentService discovery
    Contributing factor
    Unverified assumptionExcessive blast radius
    Detection

    A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

    Action items
    • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check
    Lessons

    Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

    View case
Check your cluster for this

These 3 cases involve Calico and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment3 cases

Component
Calico1Ingress controller1Service discovery1Telemetry agent1
Contributing factor
Excessive blast radius3Unverified assumption1Monitoring blind spot1No runbook1No safeguard on destructive change1
Organisation
Airbnb1CircleCI1OpenAI1
  1. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
    OpenAI2025
  2. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
  3. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019