Incident Library
Applied filters Root causeConfiguration changeComponentDatabaseSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

1 to 5 of 6

  1. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file
    Cloudflare2025
    Symptoms

    From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.

    High error rateService outageHigh latencyResponse quality degradation
    Trigger

    At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.

    Root cause
    Configuration change
    Component
    DatabaseCDN and edge network
    Contributing factor
    Unverified assumptionPartial rolloutNo staged rolloutNo configuration testsOverlapping failuresRetry storm
    Detection

    The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.

    Remediation
    • Bypassed the core proxy for Workers KV and Access so they fell back to a prior proxy version, which reduced their impact
    • Stopped the automatic creation and deployment of new Bot Management feature files
    • Inserted a known good feature file into the distribution queue and deployed it globally
    • Forced a restart of the core proxy
    • Scaled control plane concurrency to clear the backlog of dashboard logins
    • Restarted the remaining services that had entered a bad state
    Action items
    • Harden the ingestion of Cloudflare-generated configuration files the same way as user-generated input
    • Enable more global kill switches for features
    • Stop core dumps and other error reports from being able to overwhelm system resources
    • Review the failure modes for error conditions across all core proxy modules
    View case
  2. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
    Symptoms

    From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

    Service outageMonitoring outage
    Trigger

    From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

    Root cause
    Configuration change
    Component
    Message brokerDatabase
    Contributing factor
    Unrepresentative stagingNo staged rolloutIneffective rollback
    Detection

    Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

    Remediation
    • Rolled back the configuration change that over-restricted access to the metadata database
    • Identified and repaired every affected ordered subscription so the subscriptions received their backlog
    Action items
    • Enforce parity between pre-production and production environments so configuration changes are caught before production
    • Review change management so configuration changes roll out progressively according to their priority
    • Add monitoring that detects ordering metadata inconsistency
    • Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered
    View case
  3. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on them
    Microsoft Azure2024
    Symptoms

    From 21:40 UTC on 18 July 2024, virtual machines in the Azure Central US region lost access to their disks, and the services running on those VMs failed. Azure SQL Database and SQL Managed Instance showed unhealthy nodes and login failures, Cosmos DB requests failed, and service management operations failed across many Azure services.

    Storage disconnected or stale mountsService outageHigh error rateFailover failureStale endpoint routing
    Trigger

    Routine changes to the VM host fleet led to a new allow list of VM host address ranges being generated and published to all Storage scale units in Central US. Recent network configuration changes had made some of the infrastructure file servers that hold the source data unreachable from the server generating the list, so the published list was incomplete.

    Root cause
    Configuration change
    Component
    Cloud block storageDatabaseDatabase failover manager
    Contributing factor
    Unsafe failure behaviourNo staged rolloutExcessive blast radiusMonitoring blind spot
    Detection

    Service monitoring detected the drop in VM availability at 22:06 UTC while the allow list update was being published. SQL monitoring detected unhealthy nodes and login failures within one minute of SQL becoming unavailable, and monitoring flagged the availability drop for Cosmos DB, SQL DB and SQL DB Managed Instance at 22:14 UTC.

    Remediation
    • Halted the allow list update workflow worldwide
    • Paused all deployments in the Central US region
    • Paused compute buildout deployments in all regions
    • Updated the configuration of all Storage scale units in Central US, restoring storage availability by 02:55 UTC
    • Started geo-failovers for SQL databases with automatic failover policies
    • Restarted SQL gateway nodes to refresh their caches and route connections to the right nodes
    • Forced completion of SQL failovers that had not completed
    • Failed over the Cosmos DB control plane and active-passive accounts with a write region in Central US
    • Cancelled customer-triggered graceful switch region operations that blocked Cosmos DB failovers
    • Removed locks on Cosmos DB service metadata to force-complete stuck failovers
    • Repaired impacted Cosmos DB partitions for single-region accounts
    • Repaired backend nodes and failed Cosmos DB accounts back to Central US, redoing failback where it failed
    Action items
    • Make the allow list generation workflow detect incomplete source information and halt
    • Alert on storage requests rejected by allow list checks
    • Deploy allow lists one Availability Zone and storage type at a time, over 24 hours
    • Add VM health checks and an automatic stop to the allow list deployment workflow
    • Reevaluate the policy for Microsoft-managed failover of SQL failover groups
    • Fix the Cosmos DB failback workflow that broke some MongoDB drivers for MongoDB API accounts
    • Fix Cosmos DB failback for accounts with private endpoints so it does not need redoing
    • Make the storage data-plane firewall reject invalid allow lists and keep the last-known-good state
    • Improve Azure NetApp Files monitors so impacting events are detected and classified in time
    • Monitor more Azure NetApp Files service metrics to correlate similar events faster
    • Make the SQL Service Fabric cluster location change notifications reliable under load
    • Make SQL geo-failover workflows more robust so they complete
    • Remove the issues that delayed 5% of Cosmos DB failovers
    • Migrate Azure DevOps customer metadata to the right geography
    • Decouple Azure NetApp Files regional reads and writes to keep this class of issue to one region
    • Evaluate caching to reduce Azure NetApp Files reliance on stored utilization data
    • Add automatic per-partition failover for Cosmos DB multi-region active-passive accounts
    • Build the Resilient Ephemeral OS disk improvement so VMs survive storage incidents better
    View case
  4. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'
    MIT Open Learning2023
    Symptoms

    On 30 October 2023 at 3:51PM, six minutes after a configuration update was deployed to the xPro production Heroku stack, Pingdom alerted the DevOps on-call engineer, who confirmed at 3:52PM that the site was not responding.

    Service outageCredentials unavailable
    Trigger

    A configuration update applied through Salt at 3:45PM found that the Vault lease of the database credentials was about to expire and could not be renewed, so Vault generated new credentials.

    Root cause
    Configuration change
    Component
    DatabaseSecrets store
    Contributing factor
    Unreviewed defaultsChange with a delayed effectConfiguration not in code
    Detection

    Pingdom alerted the DevOps on-call engineer at 3:51PM, and the alert went through OpsGenie; Pingdom closed it automatically at 4:25PM.

    Remediation
    • Created a new database parameter group and applied it to the running RDS instance
    • Cleared the cached credentials and re-applied the Heroku configuration through Salt
    Action items
    • Move the management of xPro resources, including the RDS instance, into Pulumi, where AWS default parameter groups are not used
    Lessons

    Almost none of the configuration for this stack, the RDS instance in particular, was managed as code, and that was one of the main reasons the fix took long. Under Pulumi the instance would not have used an AWS default parameter group, and the fix would have been a change to an attached group.

    View case
  5. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
    Symptoms

    From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

    Service outageDatabase overloadHigh latencyHigh error rate
    Trigger

    A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

    Root cause
    Configuration change
    Component
    In-memory cacheDatabase
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    Google received an automated alert at 12:34 US/Pacific, one minute after the start.

    Remediation
    • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
    • Ramped Memcache traffic back up gradually while the error rate fell
    Action items
    • Reduce the number of readers of the global configuration to avoid contention during writes
    • Regionalise the configuration to limit the blast radius of similar failures
    View case
Check your cluster for this

These 6 cases involve Database and Cloud block storage. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change6 cases

Component
Database6Cloud block storage1In-memory cache1CDN and edge network1Database backups and snapshots1Database replication1Database failover manager1Message broker1Secrets store1
Contributing factor
No staged rollout3Excessive blast radius2Ineffective rollback2Unverified assumption1Configuration not in code1Unreviewed defaults1Change with a delayed effect1Monitoring blind spot1No configuration tests1No safeguard on destructive change1Overlapping failures1Partial rollout1Retry storm1Unrepresentative staging1Unsafe failure behaviour1
Organisation
Google Cloud2Cloudflare1DigitalOcean1Microsoft Azure1MIT Open Learning1
  1. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file
    Cloudflare2025
  2. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
  3. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on them
    Microsoft Azure2024
  4. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'
    MIT Open Learning2023
  5. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
  6. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted
    DigitalOcean2017