Incident Library
Applied filters Root causeConfiguration changeComponentCDN and edge networkOrganisationRust ProjectClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates
    Rust Project2023
    Symptoms

    On 25 January 2023, shortly after a deployment to the crates.io production infrastructure at 09:15 UTC, the DNS record for static.crates.io failed to resolve for an estimated 10-15 minutes. Builds failed for users because crates could not be downloaded.

    DNS resolution failures
    Trigger

    A Terraform deployment to production contained a new TLS certificate for the current CDN of static.crates.io and updated DNS records; Terraform removed the current certificate and DNS records first and recreated the records only after the new certificate had been issued.

    Root cause
    Configuration change
    Component
    TerraformAuthoritative DNS servers and recordsCDN and edge network
    Contributing factor
    No safeguard on destructive changeUnrepresentative staging
    Remediation
    • Terraform finished the deployment and created a new DNS record for static.crates.io, which ended the outage
    Action items
    • Document the process of deploying changes to production, including how to pick the Git commit and how to review the changeset
    • Deploy changes developed and tested in isolation on staging individually and in sequence to production, and add this to the documentation
    • Deploy changes to production when they are merged into the main branch, so the configuration in Git and what is deployed do not drift
    Lessons

    Deploying the TLS certificate change and the DNS record change separately would have avoided the outage. Changes tested one at a time on staging need to reach production one at a time and in order, the production deploy process needs to be written down so it can be improved, and merged changes need to be deployed promptly so Git and production do not drift apart.

    View case
Check your cluster for this

This case involves CDN and edge network and Authoritative DNS servers and records. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
CDN and edge networkAuthoritative DNS servers and recordsTerraform
Contributing factor
No safeguard on destructive changeUnrepresentative staging
Organisation
Rust Project
  1. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates
    Rust Project2023