Incident Library
Applied filters Root causeConfiguration changeComponentDatabaseSymptomHigh latencyClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases4 cases

1 to 4 of 4

  1. IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends
    Google Cloud2026
    Symptoms

    From 04:00 US/Pacific on 8 October 2026, Google Cloud Storage upload and write operations in the us-central1 region became slow and failed. Clients mostly saw HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts.

    High latencyHigh error rateLoad imbalance
    Trigger

    A configuration update to the Cloud Storage load balancing layer turned on a traffic sorting optimization that was meant to deal with local capacity constraints. The recurrence on 9 October came from an incomplete rollback of an upload planner feature.

    Root cause
    Configuration change
    Component
    Object storageDatabase
    Contributing factor
    No headroomOverlapping failures
    Detection

    Google engineering teams monitoring system telemetry detected the problem at 04:50 US/Pacific on 8 October, 50 minutes after it began.

    Remediation
    • Drained traffic from the most overloaded storage backend nodes
    • Deployed a configuration change that disabled the traffic sorting optimization
    • Rolled back a recent update to the metadata database service
    • Redirected traffic and drained affected storage components again during the recurrence
    • Fully reverted the load balancing configuration change, restoring capacity-aware routing
    • Returned the drained nodes to service once resource use was back to normal
    Action items
    • Validate load balancing logic changes in smaller regions before rolling them out to large, highly utilized regions
    • Make the storage load balancing enforce capacity constraints so traffic cannot bypass pre-spilling safeguards
    • Speed up internal data movement within highly utilized storage cells to absorb temporary load spikes
    View case
  2. IL-0561 DigitalOcean's Cloud Control Panel and Public API returned up to 25% errors in two windows as authorization timed out
    DigitalOcean2026
    Symptoms

    From 16:56 UTC on 24 August 2026, requests to DigitalOcean's Cloud Control Panel, Public API and the product management systems that depend on them failed at elevated rates. Automated monitoring flagged abnormal error rates and latency at 16:58. Running Droplets, network routing and block storage Volumes kept working.

    High error rateHigh latency
    Trigger

    A planned migration of the database layer behind DigitalOcean's identity, authentication and authorization services made one critical authorization query run much slower than its historical baseline.

    Root cause
    Configuration change
    Component
    DatabaseIdentity and access service
    Contributing factor
    Retry stormHarmful manual action
    Detection

    Automated monitoring detected abnormal error rates and latency at 16:58 UTC, two minutes after the authorization query slowed down.

    Remediation
    • Rerouted authorization traffic onto a single healthy processing path
    • Temporarily disabled hedged retries for the primary callers
    • Disabled the retry and amplification behaviour across the remaining affected services
    • Added read capacity for the slow authorization query
    Action items
    • Deploy circuit breakers in upstream services that detect a degraded downstream dependency and suppress hedged requests and retries
    • Audit the architecture so retry suppression and circuit breaking are enforced the same way in every control plane service
    • Add fast-fail load shedding so request queues do not saturate while a dependency is slow
    • Add telemetry alerts that spot queues backing up before customers are affected
    • Let critical authorization components run in a degraded mode on asynchronous, eventually consistent database replicas when the primary database is slow
    View case
  3. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file
    Cloudflare2025
    Symptoms

    From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.

    High error rateService outageHigh latencyResponse quality degradation
    Trigger

    At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.

    Root cause
    Configuration change
    Component
    DatabaseCDN and edge network
    Contributing factor
    Unverified assumptionPartial rolloutNo staged rolloutNo configuration testsOverlapping failuresRetry storm
    Detection

    The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.

    Remediation
    • Bypassed the core proxy for Workers KV and Access so they fell back to a prior proxy version, which reduced their impact
    • Stopped the automatic creation and deployment of new Bot Management feature files
    • Inserted a known good feature file into the distribution queue and deployed it globally
    • Forced a restart of the core proxy
    • Scaled control plane concurrency to clear the backlog of dashboard logins
    • Restarted the remaining services that had entered a bad state
    Action items
    • Harden the ingestion of Cloudflare-generated configuration files the same way as user-generated input
    • Enable more global kill switches for features
    • Stop core dumps and other error reports from being able to overwhelm system resources
    • Review the failure modes for error conditions across all core proxy modules
    View case
  4. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
    Symptoms

    From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

    Service outageDatabase overloadHigh latencyHigh error rate
    Trigger

    A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

    Root cause
    Configuration change
    Component
    In-memory cacheDatabase
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    Google received an automated alert at 12:34 US/Pacific, one minute after the start.

    Remediation
    • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
    • Ramped Memcache traffic back up gradually while the error rate fell
    Action items
    • Reduce the number of readers of the global configuration to avoid contention during writes
    • Regionalise the configuration to limit the blast radius of similar failures
    View case
Check your cluster for this

These 4 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change4 cases

Component
Database4In-memory cache1CDN and edge network1Identity and access service1Object storage1
Contributing factor
Overlapping failures2Retry storm2Unverified assumption1Excessive blast radius1Harmful manual action1No configuration tests1No headroom1No staged rollout1Partial rollout1Ineffective rollback1
Organisation
Google Cloud2Cloudflare1DigitalOcean1
  1. IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends
    Google Cloud2026
  2. IL-0561 DigitalOcean's Cloud Control Panel and Public API returned up to 25% errors in two windows as authorization timed out
    DigitalOcean2026
  3. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file
    Cloudflare2025
  4. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017