Graph
Cases5 cases
1 to 5 of 5
- Symptoms
On 2025-03-12 from 10:58 UTC, alerts for high backend response times came in, and the MediaWiki deployments mw-api-int, mw-parsoid and mw-jobrunner, which serve MediaWiki for non-user-facing purposes, showed high latency and database errors. The External Storage cluster, whose hosts store the compressed text of wiki page revisions, had a sharp rise in load and connections. The same happened again on 2025-03-13 from 11:15 UTC.
- Trigger
At 10:44 UTC on 12 March a Scap deployment of the PHP 8.1 rollout for mw-api-int, mw-parsoid and mw-jobrunner was run with the flag -Dbuild_mw_container_image:False; the rollout was deployed with Scap again at 11:15 UTC on 13 March.
- Root cause
- Component
- Contributing factor
- Detection
Alerts for high backend response times started at 10:58 UTC on 12 March, 14 minutes after the Scap deployment; 2 on-call engineers were paged. The MediaWikiMemcachedHighErrorRate alert that pointed at the cause fired only briefly because its log pipeline was overloaded.
- Remediation
- Deployed a patch to cirrus-streaming-updater that reduced SUP parallelism
- Reduced the concurrency of the categoryMembershipChange job in changeprop-jobqueue
- On 12 March, reverted 1126607 and 1126650 and re-ran Scap
- On 13 March, rolling-restarted changeprop-jobqueue
- Rolling-restarted mw-api-int
- Disabled the categorymembership job in changeprop-jobqueue
- Reverted 1127476 and 1127478
- Redeployed mw-api-int to pick up the revert
- Increased the number of mw-api-int replicas
- Redeployed mw-jobrunner to pick up the revert
- Redeployed mw-parsoid to pick up the revert
- Action items
- Reduce the number of messages sent through channel:Memcached during failures
- Add helm rollback functionality to Scap
- Decide whether Scap should update helmfile-defaults when -Dbuild_mw_container_image:False is set
- Consider removing envvars.inc from MediaWiki images
- Consider "Why" dashboards that give an overview of a group of related components together with relevant links to follow
- Coordinate incident responders better
- Be able to work with MediaWiki developers during incidents
- Lessons
The write-up puts the weight not on the technical root cause but on why the team could not see, even after the 12 March incident, that three MediaWiki deployments were running without Memcached. Alert dashboards showed what was happening, such as many busy PHP workers, but not why; dashboards linking components that affect each other, such as SQLBlobStore keys and External Storage, would have helped. The alert that pointed in the right direction did not stay firing and would probably have been lost among 78 others. The incident coordinator could not tell who was working on what or who had joined or left, the channels were noisy, the external impact was judged more critical than it was, and trace.wikimedia.org was not used although it might have helped.
IL-0270Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutesDiscord2023- Symptoms
On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.
- Trigger
During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.
- Root cause
- Component
- Contributing factor
- Detection
An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.
- Remediation
- Turned off sending typing events and marking messages as read to shed load from the authentication cluster
- Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
- Restarted the two zone [b] nodes one at a time
- Action items
- Double the size of the scylla-authentication cluster so each node takes half as many requests
- Rebuild scylla-authentication-b-2 with new, non-degraded disks
- Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
- Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
- Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region
- Lessons
Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.
IL-0217Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb2023- Symptoms
Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.
- Trigger
Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.
- Root cause
- Component
- Contributing factor
- Detection
Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.
- Remediation
- Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
- Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
- Failed the hard-locked database over to a replica
- Restarted all Retriever hosts so they could provide data for the cache
- Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
- Removed the circuit breaker and restarted the query hosts that failed internal checks
- Action items
- Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
- Explore ways to strengthen the cache further
- Lower the contention put on the database during update storms from new schemas
- Stabilise the performance costs of some operations in the system
- Study instrumentation and experimentation that detect and handle such edge cases better
- Lessons
Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack2022- Symptoms
Just after 6 a.m. Pacific Time on 22 February 2022, user tickets about problems connecting to Slack, problems reported by internal users, and pages and alerts to several engineering teams all arrived at almost the same moment. Many users could not connect: the Slack client could not finish booting, the step that fetches channel lists, preferences and recent conversations, and showed an error page.
- Trigger
A percentage-based rollout step upgraded the Consul agent on 25% of the fleet, followed by a slow sequential restart of the agents. Each restart on a Memcached node removed it from the Consul service catalog, so Mcrib, the cache-tier control plane, replaced it with an empty spare or flushed it when it rejoined, and the cache hit rate dropped as Slack reached its daily peak traffic.
- Root cause
- Component
- Contributing factor
- Detection
User tickets about connection problems, reports from internal users and pages and alerts to several engineering teams arrived almost at the same time, just after 6 a.m. Pacific Time.
- Remediation
- Paused the Consul agent restart operation
- Throttled client boot requests to reduce load on the database tier
- Cut the client boot rate limit again after a too-large increase and then raised it in smaller steps
- Changed the GDM membership query to read from Vitess only the data missing from Memcached
- Let the query read from replicas as well as Vitess primaries
- Raised the client boot rate limit slowly back to normal traffic levels as the caches refilled
- Action items
- Change the process for Consul rollouts so upgrades do not trigger this failure mode again
- Change the Mcrib control loop so a brief network outage or partition of cache nodes does not set off the same cascading failure
- Make the GDM membership query read from a table sharded by channel
- Check the other cache-fronted database queries for high-volume scatter queries
- Explore longer-term ways to make the caching tier more resilient
- Lessons
Slack describes performance bottlenecks, a cascading failure, dependence on a warm cache and a change as the trigger. Once the system had passed the tipping point, lower cache hit rates caused high database load that in turn kept the cache from filling, and a system in that state stays there until an outside intervention, a large cut in load or an increase in capacity, changes it. Mcrib contributed not by being wrong but by being faster than the old Consul lock scheme at replacing Memcached nodes, which increased churn in the cache tier. Client retries still added load despite backoff and jitter. A brief network partition of the cache nodes could set off the same failure, and the interaction between Consul and the caching architecture is now more widely understood.
IL-0303GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub2012- Symptoms
On Monday, September 10th, 2012, during a schema migration, the health checks on GitHub's MySQL master failed and Percona Replication Manager moved the 'active' role to another node of the new three-node cluster. That node's InnoDB buffer pool was cold and it performed poorly, its health checks failed in turn, and the role failed back to the original server. GitHub.com went down.
- Trigger
A schema migration on Monday, September 10th put more load on the master database than earlier migrations had, and Percona Replication Manager's health checks on the master failed.
- Root cause
- Component
- Contributing factor
- Detection
On the morning of September 11th, a developer told the operations team that the 'standby' node was returning incorrect query results.
- Remediation
- Enabled Pacemaker's maintenance-mode so no health checks or automatic failovers ran, and let the buffer pool warm
- Powered off the out-of-date node 'c' to end the partition
- Stopped Pacemaker and Heartbeat on nodes 'a' and 'b' and started them on node 'a' only
- Started MySQL on node 'a' in the active role
- Removed the events leaked to wrong dashboards and audited the wrongly routed repositories
- Contacted the owners of the exposed private repositories
- Action items
- Change the Pacemaker configuration so the active database role fails over only when an operations engineer starts it
- Find a way to warm the InnoDB buffer pool of slave databases so failovers do not hurt performance
- Audit the Pacemaker and Heartbeat stack, focusing on the code path behind the segfault
- Run hardware tests on the server where the segfault happened
- Lessons
GitHub concluded that automated failover is not the right strategy for its primary production database: for every failover in this incident, any member of the operations team would have said it should not happen. Failovers that must happen, in an emergency or during maintenance, should not hurt performance.
These 5 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause3
Faulty deployment1 case
- Contributing factor
- Organisation
Capacity exhaustion2 cases
- Contributing factor
Software defect2 cases
- Contributing factor
- Organisation