Cases2 cases

1 to 2 of 2

1. IL-0212 Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queries

   Slack2022

   Symptoms

   Just after 6 a.m. Pacific Time on 22 February 2022, user tickets about problems connecting to Slack, problems reported by internal users, and pages and alerts to several engineering teams all arrived at almost the same moment. Many users could not connect: the Slack client could not finish booting, the step that fetches channel lists, preferences and recent conversations, and showed an error page.

   Database overloadHigh error rateHigh latency

   Trigger

   A percentage-based rollout step upgraded the Consul agent on 25% of the fleet, followed by a slow sequential restart of the agents. Each restart on a Memcached node removed it from the Consul service catalog, so Mcrib, the cache-tier control plane, replaced it with an empty spare or flushed it when it rejoined, and the cache hit rate dropped as Slack reached its daily peak traffic.

   Root cause

   Capacity exhaustion

   Component

   DatabaseIn-memory cacheService discovery

   Contributing factor

   Dependence on a warm cacheInefficient queryAutomation amplificationRetry stormChange during peak hours

   Detection

   User tickets about connection problems, reports from internal users and pages and alerts to several engineering teams arrived almost at the same time, just after 6 a.m. Pacific Time.

   Remediation

   • Paused the Consul agent restart operation
   • Throttled client boot requests to reduce load on the database tier
   • Cut the client boot rate limit again after a too-large increase and then raised it in smaller steps
   • Changed the GDM membership query to read from Vitess only the data missing from Memcached
   • Let the query read from replicas as well as Vitess primaries
   • Raised the client boot rate limit slowly back to normal traffic levels as the caches refilled

   Action items

   • Change the process for Consul rollouts so upgrades do not trigger this failure mode again
   • Change the Mcrib control loop so a brief network outage or partition of cache nodes does not set off the same cascading failure
   • Make the GDM membership query read from a table sharded by channel
   • Check the other cache-fronted database queries for high-volume scatter queries
   • Explore longer-term ways to make the caching tier more resilient

   Lessons

   Slack describes performance bottlenecks, a cascading failure, dependence on a warm cache and a change as the trigger. Once the system had passed the tipping point, lower cache hit rates caused high database load that in turn kept the cache from filling, and a system in that state stays there until an outside intervention, a large cut in load or an increase in capacity, changes it. Mcrib contributed not by being wrong but by being faster than the old Consul lock scheme at replacing Memcached nodes, which increased churn in the cache tier. Client retries still added load despite backoff and jitter. A brief network partition of the cache nodes could set off the same failure, and the interaction between Consul and the caching architecture is now more widely understood.

   View case
2. IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster access

   Elastic2019

   Symptoms

   On 18 January 2019 from 09:09 UTC the leader of the three-node Apache ZooKeeper ensemble behind Elastic Cloud in AWS eu-west-1 (Ireland) ran at heavy CPU load, and its clients lost their connections. From about 10:08 UTC customers in the region had severely degraded access to their Elasticsearch and Kibana deployments for roughly 3 hours, and for about 20 minutes every deployment in the region was unavailable.

   Database overloadService outageOOMKilled

   Trigger

   At 09:09 UTC the leader server of the ZooKeeper ensemble in the eu-west-1 coordination layer hit a high CPU threshold, which caused client disconnections.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discovery

   Contributing factor

   Retry stormUnpatched known bug

   Detection

   The on-call SREs received alerts about service degradation in the coordination layer at 09:35 UTC, after the high CPU on the ZooKeeper leader had caused client disconnections, and started a production incident.

   Remediation

   • Scaled the proxy layer up and out to take the surging client traffic hitting the load balancers
   • Triggered a leader election in the ZooKeeper ensemble and stabilised it

   Action items

   • Replace Curator's TreeCache with TreeWatcher across all Elasticsearch Service providers and regions
   • Contribute TreeWatcher upstream to the Apache Curator project
   • Roll out a new proxy layer that keeps the routing table in a versioned local file, so it can route traffic while the ZooKeeper connection is lost or flaky

   View case

By root cause2