Cases7 cases

1 to 5 of 7

1. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file

   Cloudflare2025

   Symptoms

   From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.

   High error rateService outageHigh latencyResponse quality degradation

   Trigger

   At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.

   Root cause

   Configuration change

   Component

   DatabaseCDN and edge network

   Contributing factor

   Unverified assumptionPartial rolloutNo staged rolloutNo configuration testsOverlapping failuresRetry storm

   Detection

   The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.

   Remediation

   • Bypassed the core proxy for Workers KV and Access so they fell back to a prior proxy version, which reduced their impact
   • Stopped the automatic creation and deployment of new Bot Management feature files
   • Inserted a known good feature file into the distribution queue and deployed it globally
   • Forced a restart of the core proxy
   • Scaled control plane concurrency to clear the backlog of dashboard logins
   • Restarted the remaining services that had entered a bad state

   Action items

   • Harden the ingestion of Cloudflare-generated configuration files the same way as user-generated input
   • Enable more global kill switches for features
   • Stop core dumps and other error reports from being able to overwhelm system resources
   • Review the failure modes for error conditions across all core proxy modules

   View case
2. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster

   Wikimedia Foundation2025

   Symptoms

   On 2025-03-12 from 10:58 UTC, alerts for high backend response times came in, and the MediaWiki deployments mw-api-int, mw-parsoid and mw-jobrunner, which serve MediaWiki for non-user-facing purposes, showed high latency and database errors. The External Storage cluster, whose hosts store the compressed text of wiki page revisions, had a sharp rise in load and connections. The same happened again on 2025-03-13 from 11:15 UTC.

   Database overloadHigh latencyHigh error rateLog floodMonitoring outage

   Trigger

   At 10:44 UTC on 12 March a Scap deployment of the PHP 8.1 rollout for mw-api-int, mw-parsoid and mw-jobrunner was run with the flag -Dbuild_mw_container_image:False; the rollout was deployed with Scap again at 11:15 UTC on 13 March.

   Root cause

   Faulty deployment

   Component

   DatabaseIn-memory cacheConfigMaps and SecretsHelmCI system

   Contributing factor

   Unverified assumptionDependence on a warm cacheSignal buried in noise

   Detection

   Alerts for high backend response times started at 10:58 UTC on 12 March, 14 minutes after the Scap deployment; 2 on-call engineers were paged. The MediaWikiMemcachedHighErrorRate alert that pointed at the cause fired only briefly because its log pipeline was overloaded.

   Remediation

   • Deployed a patch to cirrus-streaming-updater that reduced SUP parallelism
   • Reduced the concurrency of the categoryMembershipChange job in changeprop-jobqueue
   • On 12 March, reverted 1126607 and 1126650 and re-ran Scap
   • On 13 March, rolling-restarted changeprop-jobqueue
   • Rolling-restarted mw-api-int
   • Disabled the categorymembership job in changeprop-jobqueue
   • Reverted 1127476 and 1127478
   • Redeployed mw-api-int to pick up the revert
   • Increased the number of mw-api-int replicas
   • Redeployed mw-jobrunner to pick up the revert
   • Redeployed mw-parsoid to pick up the revert

   Action items

   • Reduce the number of messages sent through channel:Memcached during failures
   • Add helm rollback functionality to Scap
   • Decide whether Scap should update helmfile-defaults when -Dbuild_mw_container_image:False is set
   • Consider removing envvars.inc from MediaWiki images
   • Consider "Why" dashboards that give an overview of a group of related components together with relevant links to follow
   • Coordinate incident responders better
   • Be able to work with MediaWiki developers during incidents

   Lessons

   The write-up puts the weight not on the technical root cause but on why the team could not see, even after the 12 March incident, that three MediaWiki deployments were running without Memcached. Alert dashboards showed what was happening, such as many busy PHP workers, but not why; dashboards linking components that affect each other, such as SQLBlobStore keys and External Storage, would have helped. The alert that pointed in the right direction did not stay firing and would probably have been lost among 78 others. The incident coordinator could not tell who was working on what or who had joined or left, the channels were noisy, the external impact was judged more critical than it was, and trace.wikimedia.org was not used although it might have helped.

   View case
3. IL-0312 Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went down

   Gadget2024

   Symptoms

   At 11:25 AM EST on 13 December 2024, Gadget restarted its main PostgreSQL server in place to raise max_connections. Postgres shut down but did not start again, and every request to Gadget applications failed, including frontend requests, GraphQL API requests, webhook processing and syncs.

   Service outageHigh latency

   Trigger

   An emergency in-place restart of the main Postgres server at 11:25 AM EST, decided at 11:00 AM, to raise max_connections, which Postgres cannot change without a full restart.

   Root cause

   Capacity exhaustion

   Component

   DatabaseLinux kernelDatabase proxy and connection pooler

   Contributing factor

   Unverified assumptionUnrepresentative stagingAutomation amplificationExcessive blast radius

   Detection

   The team carrying out the planned restart saw at 11:26 AM EST that Postgres had shut down and did not start again.

   Remediation

   • Restarted the database server with a hardware reset, which cleared the fragmented memory so Postgres could start
   • Provisioned almost 4x the compute after the autoscalers had scaled it down during the outage

   Action items

   • Use only the replica-failover approach for disruptive database operations such as restarts
   • Run a more aggressive capacity planning process before Black Friday / Cyber Monday, preparing for a 6x increase in load and proving it with load tests well in advance
   • Announce planned downtime ahead of time on the status page and community Discord, and make the operating procedures tell planned maintenance apart from service disruptions
   • Finish the database sharding project so apps can use different databases and configuration changes can go through canary and blue/green setups
   • Finish building a zero-downtime database failover system
   • Set much higher scaling floors on the compute autoscalers

   Lessons

   Gadget names three deeper factors. Choosing the in-place restart put the database into a state the team had not seen before and was not confident in. Capacity planning for Black Friday / Cyber Monday assumed a doubling of traffic instead of the almost 5x that came, so settings such as max_connections stopped fitting and had to be changed in an emergency instead of in a maintenance window practised in staging. And there was no way to test database configuration changes in a production-like environment: integration tests run against a synthetic Postgres database that never goes through the months of production load behind the memory fragmentation or the memory usage issues.

   View case
4. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes

   Discord2023

   Symptoms

   On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

   Service outageDatabase overloadHigh latency

   Trigger

   During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm

   Detection

   An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

   Remediation

   • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
   • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
   • Restarted the two zone [b] nodes one at a time

   Action items

   • Double the size of the scylla-authentication cluster so each node takes half as many requests
   • Rebuild scylla-authentication-b-2 with new, non-degraded disks
   • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
   • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
   • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region

   Lessons

   Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

   View case
5. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case

By root cause6