Incident Library
Applied filters Root causeFaulty deploymentSymptomService outageFactorIneffective rollbackClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0328 A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"
    CircleCI2021
    Symptoms

    From about 18:50 to 19:43 UTC on Monday, November 8, 2021, CircleCI customer jobs on all executors were blocked in a "not running" state. Errors and alerts appeared right after a deploy that changed a field type in the database of the job distribution service.

    Service outageHigh latencyScale-up failureCloud API rate limit exhausted
    Trigger

    At 18:50 UTC a deploy changed the type of a field in the PostgreSQL database of the job distribution service, in a way that was not backwards compatible.

    Root cause
    Faulty deployment
    Component
    DatabaseNode pool
    Contributing factor
    Ineffective rollbackExcessive blast radiusGap in automated tests
    Detection

    Errors and alerts related to the distribution failure appeared immediately after the 18:50 UTC deploy.

    Remediation
    • Rolled back the field type change, which did not clear the errors
    • Wrote, built and manually deployed a change that made the distribution system ignore the field
    • Manually scaled the primary compute cluster for the expected thundering herd of work
    • Monitored until processing was normal and resolved the incident
    Action items
    • Fix the gaps found in the testing strategy
    • Change how the distributor reacts to unexpected data so that this kind of deploy should not affect all jobs
    • Make recovery faster when unforeseen failures occur
    View case
  2. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN
    bunny.net2021
    Symptoms

    On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

    Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance
    Trigger

    At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

    Root cause
    Faulty deployment
    Component
    Authoritative DNS servers and recordsCDN and edge networkObject storage
    Contributing factor
    Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action
    Detection

    About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

    Remediation
    • Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
    • Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
    • Migrated the deployment systems and files to a third-party cloud storage service
    • Rewired the deployment system and DNS software to the new storage and deployed
    • Synced the databases and deployed the newest file sets through the third-party service
    Action items
    • Phase out the BinaryPack library as a hot-fix
    • Test third-party libraries more extensively
    • Migrate the internal APIs to an independent third-party service
    • Find how to stop non-critical software from becoming a single point of failure across clusters
    • Make the DNS system keep a local copy of all backup data with automatic failure detection
    Lessons

    bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

    View case
Check your cluster for this

These 2 cases involve CDN and edge network and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment2 cases

Component
CDN and edge network1Database1Authoritative DNS servers and records1Node pool1Object storage1
Contributing factor
Ineffective rollback2Automation amplification1Excessive blast radius1Circular dependency1Harmful manual action1Single point of failure1Gap in automated tests1
Organisation
bunny.net1CircleCI1
  1. IL-0328 A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"
    CircleCI2021
  2. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN
    bunny.net2021