IL-0363 Rust Project · 2023-01-25networking

static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates

The Rust infrastructure team manages its infrastructure with Terraform and had recently separated the staging and production environments of crates.io so that each can be deployed on its own. The staging environment was used to build and test a second CDN for static.crates.io, and the rollout to production was scheduled and announced for 25 January 2023. The production deployment carried two changes that had each been developed, deployed and tested separately on staging: a new TLS certificate for the current CDN and updated DNS records. In production, Terraform first deleted the current certificate and the DNS records, then spent about 10 minutes issuing the new certificate, during which static.crates.io had no DNS record, and only then recreated the records. The outage ended by itself when Terraform finished.

The team names two reasons the changes went out together. It was one of the first uses of the new environment tooling, which locks production to a specific Git commit, and the team set it to the latest commit on master as in the past, so several changes were applied at once. And production had drifted away from staging because changes were not deployed when they were merged, partly because the team was reluctant to deploy crates.io several times without announcing each change to the community first.

Severity: –
Scope: Name resolution of static.crates.io; builds failed because crates could not be downloaded
Users affected: –
Region: –
Time to detect: –
Time to resolve: 25m

Symptoms

On 25 January 2023, shortly after a deployment to the crates.io production infrastructure at 09:15 UTC, the DNS record for static.crates.io failed to resolve for an estimated 10-15 minutes. Builds failed for users because crates could not be downloaded.

The record started to propagate again around 09:30 UTC and traffic was back to normal levels by 09:40 UTC. For some users the failures lasted a few minutes longer because of caches in their DNS servers.

Symptom DNS resolution failures

Root cause

Root cause Configuration change

Component Terraform Authoritative DNS servers and records CDN and edge network

Contributing factor No safeguard on destructive change Unrepresentative staging

Trigger: A Terraform deployment to production contained a new TLS certificate for the current CDN of static.crates.io and updated DNS records; Terraform removed the current certificate and DNS records first and recreated the records only after the new certificate had been issued.

Detection

–

Remediation

Terraform finished the deployment and created a new DNS record for static.crates.io, which ended the outage

Action items

Document the process of deploying changes to production, including how to pick the Git commit and how to review the changeset
Deploy changes developed and tested in isolation on staging individually and in sequence to production, and add this to the documentation
Deploy changes to production when they are merged into the main branch, so the configuration in Git and what is deployed do not drift

Lessons

Deploying the TLS certificate change and the DNS record change separately would have avoided the outage. Changes tested one at a time on staging need to reach production one at a time and in order, the production deploy process needs to be written down so it can be improved, and merged changes need to be deployed promptly so Git and production do not drift apart.

Sources

https://blog.rust-lang.org/inside-rust/2023/02/08/dns-outage-portmortem/ · blog · archived copy

Related cases

IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedMIT Open Learning · 2026 · sev1
same root cause · shared Authoritative DNS servers and records, CDN and edge network
IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature fileCloudflare · 2025
same root cause · shared CDN and edge network
IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster2i2c · 2025
same root cause · shared Authoritative DNS servers and records
IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutesRailway · 2024
same root cause · shared Terraform
IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hoursMicrosoft Azure · 2024
same root cause · shared CDN and edge network
IL-0533 Names in chinacloudapp.cn and chinacloudsites.cn stopped resolving as DNS caches expired, breaking Azure China servicesMicrosoft Azure · 2024
same root cause · shared Authoritative DNS servers and records
IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hoursSquare · 2023
same root cause · shared Authoritative DNS servers and records
IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023
same root cause · shared Terraform
IL-0549 Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removedAtlassian · 2023
same root cause · shared Authoritative DNS servers and records
IL-0541 Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to themTwingate · 2023
same root cause · shared Terraform
IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1
same root cause · shared Terraform
IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021
same root cause · shared Terraform
IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster serviceDFDS · 2020
same root cause · shared Terraform
IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019
same root cause · shared Terraform

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Terraform, Authoritative DNS servers and records, CDN and edge network among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0363
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0363/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.