Graph
Cases4 cases
1 to 4 of 4
- Symptoms
At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.
- Trigger
The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.
- Root cause
- Component
- Contributing factor
- Detection
Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.
- Remediation
- Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate
- Action items
- Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
- Deploy the Pulumi update for the xPRO application in Production
- Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
- Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
- Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires
- Lessons
Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.
IL-0283Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature fileCloudflare2025- Symptoms
From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.
- Trigger
At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.
- Root cause
- Component
- Contributing factor
- Detection
The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.
- Remediation
- Bypassed the core proxy for Workers KV and Access so they fell back to a prior proxy version, which reduced their impact
- Stopped the automatic creation and deployment of new Bot Management feature files
- Inserted a known good feature file into the distribution queue and deployed it globally
- Forced a restart of the core proxy
- Scaled control plane concurrency to clear the backlog of dashboard logins
- Restarted the remaining services that had entered a bad state
- Action items
- Harden the ingestion of Cloudflare-generated configuration files the same way as user-generated input
- Enable more global kill switches for features
- Stop core dumps and other error reports from being able to overwhelm system resources
- Review the failure modes for error conditions across all core proxy modules
IL-0339Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hoursMicrosoft Azure2024- Symptoms
From 11:45 UTC on 30 July 2024, a subset of customers got intermittent connection errors, timeouts and latency spikes when connecting to Microsoft services that use Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The Azure Portal team saw the first service degradation at 11:47 UTC.
- Trigger
Around 11:45 UTC the Azure network DDoS protection service was disengaging after a routine DDoS mitigation and returning traffic to its default routing to AFD, but the routes at one site in Europe could not be updated because the network control plane could not reach that site during a local power outage.
- Root cause
- Component
- Contributing factor
- Detection
The Azure Portal team detected service degradation at 11:47 UTC and began to investigate. At 12:10 UTC network monitoring correlated the Portal incident to a network issue at one site in Europe, and networking engineers joined.
- Remediation
- Rerouted traffic under the standard congestion playbook, which removed most of the impact by 13:58 UTC
- Ordered the network control plane to reissue its commands to the misbehaving device, which could not be reached
- Removed the device that kept attracting traffic from the network and scanned the network for other issues
- Action items
- Add the missing configuration on network devices so a DDoS mitigation issue in one geography cannot spread to another
- Improve validation and monitoring in the Azure network to detect invalid configurations
- Improve monitoring for when the DDoS protection service is unreachable from the control plane but still serving traffic
IL-0363static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project2023- Symptoms
On 25 January 2023, shortly after a deployment to the crates.io production infrastructure at 09:15 UTC, the DNS record for static.crates.io failed to resolve for an estimated 10-15 minutes. Builds failed for users because crates could not be downloaded.
- Trigger
A Terraform deployment to production contained a new TLS certificate for the current CDN of static.crates.io and updated DNS records; Terraform removed the current certificate and DNS records first and recreated the records only after the new certificate had been issued.
- Root cause
- Component
- Contributing factor
- Remediation
- Terraform finished the deployment and created a new DNS record for static.crates.io, which ended the outage
- Action items
- Document the process of deploying changes to production, including how to pick the Git commit and how to review the changeset
- Deploy changes developed and tested in isolation on staging individually and in sequence to production, and add this to the documentation
- Deploy changes to production when they are merged into the main branch, so the configuration in Git and what is deployed do not drift
- Lessons
Deploying the TLS certificate change and the DNS record change separately would have avoided the outage. Changes tested one at a time on staging need to reach production one at a time and in order, the production deploy process needs to be written down so it can be improved, and merged changes need to be deployed promptly so Git and production do not drift apart.
These 4 cases involve CDN and edge network and Authoritative DNS servers and records. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause1
Configuration change4 cases
- Component
- Contributing factor
IL-0512xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedIL-0283Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature fileCloudflare2025IL-0339Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hoursMicrosoft Azure2024IL-0363static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project2023