Incident Library
Applied filters Root causeConfiguration changeComponentAuthoritative DNS servers and recordsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

3 to 6 of 6

  1. IL-0533 Names in chinacloudapp.cn and chinacloudsites.cn stopped resolving as DNS caches expired, breaking Azure China services
    Microsoft Azure2024
    Symptoms

    From 03:45 CST on 23 April 2024, FQDNs in two system domains of Azure in China, chinacloudapp.cn and chinacloudsites.cn, gradually stopped resolving, and customers in the Azure China regions had trouble connecting to many Azure services and managing resources. Internal monitors alerted engineers to the degradation at 04:37 CST.

    DNS resolution failuresHigh error rateMonitoring outage
    Trigger

    At 03:45 CST a configuration change made through a domain registrar, as a step of a domain decommissioning workflow, updated the nameservers of chinacloudapp.cn and chinacloudsites.cn to a set of inactive servers.

    Root cause
    Configuration change
    Component
    Authoritative DNS servers and records
    Contributing factor
    Unverified assumptionCache-masked failure
    Detection

    Internal monitors alerted engineers to the degradation in the Azure China Cloud at 04:37 CST and created an incident, which the engineering team acknowledged at 04:39 CST.

    Remediation
    • Reverted the two domains to their previous working nameservers
    Action items
    • Suspended further runs of the domain lifecycle process until the management tool covers Azure in China
    • Make the validation process of domain lifecycle management include signals from all cloud regions
    • Add validations that get nameserver information by resolving the zone directly over the Internet
    View case
  2. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours
    Square2023
    Symptoms

    From 1:54 PM ET on 7 September 2023, Square products and services were unavailable, and at 2:02 PM ET engineers were notified of infrastructure issues and began investigating DNS. Square Online websites stayed up, but their customers could not make payments.

    DNS resolution failuresService outage
    Trigger

    At 11:04 AM ET a host-based firewall rule change was deployed to enable region communication, which increased the size of the firewall rules on nodes. Combined with a DNS service upgrade, it put unexpected load on the internal DNS servers. The timeline also lists a DNS zone change at 1:56 PM ET.

    Root cause
    Configuration change
    Component
    Authoritative DNS servers and recordsNode iptables rules
    Contributing factor
    No staged rolloutCache-masked failureCircular dependency
    Detection

    Engineers were notified of infrastructure issues at 2:02 PM ET, eight minutes after services became unavailable.

    Remediation
    • Shed network traffic to the DNS servers and started bringing up new DNS servers by hand
    • Deployed networking changes by hand to re-enable the authorization and access services
    • Deployed the networking changes by hand to all data centers
    • Rebuilt the DNS servers
    • Built a new configuration from the reverted firewall ruleset and pushed it to the DNS hosts
    Action items
    • Move the DNS infrastructure onto isolated infrastructure
    • Add monitoring and optimizations for critical networking infrastructure
    • Optimize dependencies between the deploy and platform infrastructure where feasible
    • Expand support for Offline Mode and improve communication about when it is available
    View case
  3. IL-0549 Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removed
    Atlassian2023
    Symptoms

    From 02:36 UTC on 15 May 2023, Atlassian products with services hosted in the us-west-1 region could not store or retrieve data assets such as media, attachments and build artifacts. Automated monitoring detected it within three minutes, and the status page reported intermittent errors.

    High error rateStale endpoint routing
    Trigger

    An internal service was migrated to a new network segment as part of an initiative to harden the partitions between network segments, and the old network stack was then removed.

    Root cause
    Configuration change
    Component
    Authoritative DNS servers and records
    Contributing factor
    Unverified assumptionUnrepresentative staging
    Detection

    Automated monitoring detected the incident within three minutes.

    Remediation
    • Rolled back the network migration change, returning Atlassian systems to a known good state
    Action items
    • Review the systems that decommission service stacks and check that customer traffic is no longer served before a stack is decommissioned
    • Add steps to the service network migration process that identify associated DNS records needing attention
    Lessons

    Atlassian said the migration used blue/green deployment to limit this kind of risk, but adequate verification was not run before the old stack was decommissioned, and moving services between network segments is rare and hard to replicate in test.

    View case
  4. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates
    Rust Project2023
    Symptoms

    On 25 January 2023, shortly after a deployment to the crates.io production infrastructure at 09:15 UTC, the DNS record for static.crates.io failed to resolve for an estimated 10-15 minutes. Builds failed for users because crates could not be downloaded.

    DNS resolution failures
    Trigger

    A Terraform deployment to production contained a new TLS certificate for the current CDN of static.crates.io and updated DNS records; Terraform removed the current certificate and DNS records first and recreated the records only after the new certificate had been issued.

    Root cause
    Configuration change
    Component
    TerraformAuthoritative DNS servers and recordsCDN and edge network
    Contributing factor
    No safeguard on destructive changeUnrepresentative staging
    Remediation
    • Terraform finished the deployment and created a new DNS record for static.crates.io, which ended the outage
    Action items
    • Document the process of deploying changes to production, including how to pick the Git commit and how to review the changeset
    • Deploy changes developed and tested in isolation on staging individually and in sequence to production, and add this to the documentation
    • Deploy changes to production when they are merged into the main branch, so the configuration in Git and what is deployed do not drift
    Lessons

    Deploying the TLS certificate change and the DNS record change separately would have avoided the outage. Changes tested one at a time on staging need to reach production one at a time and in order, the production deploy process needs to be written down so it can be improved, and merged changes need to be deployed promptly so Git and production do not drift apart.

    View case
Check your cluster for this

These 6 cases involve Authoritative DNS servers and records and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change6 cases

Component
Authoritative DNS servers and records6CDN and edge network2cert-manager1Node iptables rules1Terraform1
Contributing factor
Unverified assumption3Cache-masked failure3Unrepresentative staging2Circular dependency1Unreviewed defaults1Change with a delayed effect1Monitoring blind spot1No configuration tests1No safeguard on destructive change1No staged rollout1
Organisation
2i2c1Atlassian1Microsoft Azure1MIT Open Learning1Rust Project1Square1
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster
    2i2c2025
  3. IL-0533 Names in chinacloudapp.cn and chinacloudsites.cn stopped resolving as DNS caches expired, breaking Azure China services
    Microsoft Azure2024
  4. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours
    Square2023
  5. IL-0549 Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removed
    Atlassian2023
  6. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates
    Rust Project2023