IL-0212 Slack · 2022-02-22database
Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queries
Slack stores its data in Vitess, a horizontal scaling system for MySQL with logical databases called keyspaces, and caches frequently used data in Memcached. Mcrouter routes cache requests by consistent hashing over a list of healthy Memcached hosts, and Mcrib, a fairly new control plane, generates that list from the Consul service catalog: when a Memcached instance disappears from Consul, Mcrib promotes a spare, and a node that rejoins is flushed before it serves again. During client boot Slack fetches the members of each GDM the user is in. That data was read from a keyspace sharded by user, so finding the members of one GDM meant a scatter query to every shard. The data is immutable and cached by channel ID with a long TTL, so the query was normally almost always served from the cache.
On 22 February a Consul agent upgrade step covered 25% of the fleet, after two 25% steps the week before had passed without trouble. As Memcached nodes were restarted one after another at the daily traffic peak, Mcrib kept replacing and flushing them and the cache hit rate fell. Because even one channel missing from the cache meant querying every shard, nearly every booting client now needed a query on every shard, and the keyspace was overwhelmed. Timed-out queries could not refill the cache, and client retries added load, so the system stayed in a cascading failure even after the Consul restarts were paused.
Responders throttled client boot requests to cut the load on the datastores, which restored more normal service for users with booted clients. The query was changed to read from Vitess only the data missing from Memcached instead of every shard on each miss, and, because the data tolerates staleness, to read from replicas as well as primaries. The caches refilled, hit rates rose and database load fell, and the boot rate limit was raised slowly back to normal traffic levels until service was fully restored.
- Severity
- –
- Scope
- Slack client boot: many users could not connect to Slack, and requests from users with booted clients also failed until boot requests were throttled
- Users affected
- many Slack users, including internal users
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
Just after 6 a.m. Pacific Time on 22 February 2022, user tickets about problems connecting to Slack, problems reported by internal users, and pages and alerts to several engineering teams all arrived at almost the same moment. Many users could not connect: the Slack client could not finish booting, the step that fetches channel lists, preferences and recent conversations, and showed an error page.
Responders saw more load than usual on parts of the database system. One Vitess keyspace, which holds channel membership sharded by user, was severely overloaded, and a large share of its load came from a query that lists Group Direct Message (GDM) conversations by user. That query normally sits behind the Memcached cache tier, so the load suggested that something was wrong with the caches. Because the table was sharded by user, finding the members of a GDM took a scatter query to every shard, and with a large part of the cache empty nearly every user needed such a query to succeed on every shard. Most queries to the channel membership tables sharded by user were timing out, which also meant the cache could not be refilled.
Client boot requests took much longer than usual and often failed, and these slow requests exhausted resources in the database tier, so requests from users whose clients had already booted failed too. Throttling client boot requests reduced the errors and gave users with booted clients more normal service. The first attempt to raise the boot limit went too far: database load rose past sustainable levels and failed requests climbed again, so the limit had to be cut and then raised in smaller steps.
Early on, responders paused the Consul agent restarts that were in progress, before anyone suspected them, but by then this made no difference. The database read load grew superlinearly with the share of cache misses, and client retries, even with exponential backoff and jitter, added to it, so the cache and database tiers stayed in a cascading failure until load was reduced and the query was changed.
Root cause
Root cause Capacity exhaustion
- Trigger
- A percentage-based rollout step upgraded the Consul agent on 25% of the fleet, followed by a slow sequential restart of the agents. Each restart on a Memcached node removed it from the Consul service catalog, so Mcrib, the cache-tier control plane, replaced it with an empty spare or flushed it when it rejoined, and the cache hit rate dropped as Slack reached its daily peak traffic.
Detection
User tickets about connection problems, reports from internal users and pages and alerts to several engineering teams arrived almost at the same time, just after 6 a.m. Pacific Time.
Remediation
- Paused the Consul agent restart operation
- Throttled client boot requests to reduce load on the database tier
- Cut the client boot rate limit again after a too-large increase and then raised it in smaller steps
- Changed the GDM membership query to read from Vitess only the data missing from Memcached
- Let the query read from replicas as well as Vitess primaries
- Raised the client boot rate limit slowly back to normal traffic levels as the caches refilled
Action items
- Change the process for Consul rollouts so upgrades do not trigger this failure mode again
- Change the Mcrib control loop so a brief network outage or partition of cache nodes does not set off the same cascading failure
- Make the GDM membership query read from a table sharded by channel
- Check the other cache-fronted database queries for high-volume scatter queries
- Explore longer-term ways to make the caching tier more resilient
Lessons
Slack describes performance bottlenecks, a cascading failure, dependence on a warm cache and a change as the trigger. Once the system had passed the tipping point, lower cache hit rates caused high database load that in turn kept the cache from filling, and a system in that state stays there until an outside intervention, a large cut in load or an increase in capacity, changes it. Mcrib contributed not by being wrong but by being faster than the old Consul lock scheme at replacing Memcached nodes, which increased churn in the cache tier. Client retries still added load despite backoff and jitter. A brief network partition of the cache nodes could set off the same failure, and the interaction between Consul and the caching architecture is now more widely understood.
Sources
Related cases
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared In-memory cache, Database
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared In-memory cache, Database
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared In-memory cache, Database
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared In-memory cache, Database
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Service discovery, Database
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared In-memory cache, Database
IL-0432AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limitHello Inc. · 2026same root cause · shared In-memory cache
IL-0450Oracle RAC commits behind order APIs kept waiting over 0.1 s on log file sync while average host CPU had headroomToss Securities · 2026same root cause · shared Database
IL-0553CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutesCircleCI · 2026same root cause · shared Database
IL-0379InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026same root cause · shared Database
IL-0353Fly.io edge proxies in US west coast regions were overloaded and flapping, with lazy loader delays of secondsFly.io · 2026same root cause · shared Service discovery
IL-0290GitLab.com's Redis Sidekiq cluster ran out of memory and lost quorum, causing 500 errors and a 12 million job backlogGitLab · 2026 · sev1same root cause · shared In-memory cache
IL-0309Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hoursPreset · 2026same root cause · shared Database
IL-0435Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flatCyberAgent · 2026same root cause · shared Database
IL-0497Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passedButtondown · 2026same root cause · shared Database
IL-0288ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026same root cause · shared Database
IL-0350Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud · 2026same root cause · shared Database
IL-0305Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026same root cause · shared Database
IL-0390PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalledPostHog · 2025same root cause · shared Database
IL-058129CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week saleMUSINSA · 2025same root cause · shared In-memory cache
IL-0287One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operationsMongoDB · 2025same root cause · shared Database
IL-0557Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hoursStorj · 2025same root cause · shared Database
IL-0067Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restartBC Government · 2025 · sev1same root cause · shared In-memory cache
IL-0551Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a dayAtlassian · 2025same root cause · shared Database
IL-0298Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disksWikimedia Foundation · 2025same root cause · shared Database
IL-0312Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went downGadget · 2024same root cause · shared Database
IL-0447An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baselineevery · 2024same root cause · shared Database
IL-0375MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a databaseMongoDB · 2024same root cause · shared Database
IL-0110A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Database
IL-0550Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pendingAtlassian · 2024same root cause · shared Database
IL-0270Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutesDiscord · 2023same root cause · shared Database
IL-0326A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hoursMailPace · 2023same root cause · shared Database
IL-0546Buildkite jobs could not go to newly connected agents once agent IDs passed a 32-bit integer foreign key's limitBuildkite · 2023same root cause · shared Database
IL-0453Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forklessHyperconnect · 2022same root cause · shared In-memory cache
IL-0500UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up2i2c · 2022same root cause · shared Database
IL-0301Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice37signals · 2022same root cause · shared Database
IL-0302Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022GitHub · 2022same root cause · shared Database
IL-0456Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQLInflab · 2022same root cause · shared Database
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021same root cause · shared Database
IL-0496Ably's Cassandra hit 100% CPU in every region and queries timed out globally, and the load returned on two more eveningsAbly · 2020same root cause · shared Database
IL-0443A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019same root cause · shared Database
IL-0037Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Database
IL-0211A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hoursMailchimp · 2019same root cause · shared Database
IL-0064CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Database
IL-0300Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,64737signals · 2018same root cause · shared Database
IL-0334Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went downInstapaper · 2017same root cause · shared Database
IL-0333A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com downBuildkite · 2016same root cause · shared Database
IL-0520logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessionscloud.gov · 2016same root cause · shared In-memory cache
IL-0296Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customersTransloadit · 2016same root cause · shared In-memory cache
IL-0282DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata serviceAmazon Web Services · 2015same root cause · shared Database
IL-0275Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hoursParse.ly · 2015same root cause · shared Service discovery
IL-0383PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming eventsPagerDuty · 2014 · sev1same root cause · shared Database
IL-0320Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0212
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0212/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.