Incident Library

Organisation

CircleCI

CircleCI · circleci

1 case · 2021

Root cause Faulty deployment
Component CalicoIngress controller
Contributing factor Excessive blast radiusMonitoring blind spotNo runbook
Symptom Resources deletedService outage

Cases

  1. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.comCircleCI · 2021

    Root cause Faulty deployment