Incident Library
Applied filters Root causeFaulty deploymentSymptomService outageFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
    Symptoms

    At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.

    Service outageResources deleted
    Trigger

    An update deployed to a test Kubernetes cluster also removed Traefik and Calico resources from CircleCI's primary production cluster.

    Root cause
    Faulty deployment
    Component
    Ingress controllerCalico
    Contributing factor
    Excessive blast radiusNo runbookMonitoring blind spot
    Detection

    CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.

    Remediation
    • restored Traefik in the production cluster
    • restored Calico in the production cluster, after which pods served customer requests again
    Action items
    • Improve network-level documentation and runbooks to cover every layer of the Kubernetes networks
    • Improve monitoring and alerting for the network infrastructure
    • Add guardrails that isolate the production and staging clusters so they cannot interfere with each other
    Lessons

    A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again. Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.

    View case
  2. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
    Symptoms

    An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

    Probe failuresService outage
    Trigger

    A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

    Root cause
    Faulty deployment
    Component
    Telemetry agentService discovery
    Contributing factor
    Unverified assumptionExcessive blast radius
    Detection

    A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

    Action items
    • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check
    Lessons

    Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

    View case
Check your cluster for this

These 2 cases involve Calico and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment2 cases

Component
Calico1Ingress controller1Service discovery1Telemetry agent1
Contributing factor
Excessive blast radius2Unverified assumption1Monitoring blind spot1No runbook1
Organisation
Airbnb1CircleCI1
  1. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
  2. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019