Cases4 cases

1 to 4 of 4

1. IL-0346 Render's routing service returned intermittent 404s for web services and static sites in Oregon

   Render2025

   Symptoms

   On 5 November 2025, some web services and static sites hosted on Render in the Oregon region answered part of their requests with 404 responses. Every newly created service in the affected clusters got 404 responses, and updates to existing services were slow to take effect.

   High error rate

   Trigger

   On 5 November 2025 Render cleaned up feature flags that no longer appeared in code, including a flag that the metadata service still read, so the metadata service went back to its older, less memory-efficient way of sending data to the routing service.

   Root cause

   Configuration change

   Component

   Cloud load balancerService discovery

   Contributing factor

   Partial rolloutNo health probesAutomation amplification

   Remediation

   • Increased the memory available to the metadata service, a change later reverted
   • Re-enabled the feature flag for the more efficient data transfer between the routing and metadata services for a time; the flag was later removed
   • Deployed the metadata service so that it no longer relied on the feature flag

   Action items

   • Enhanced monitoring of the metadata service so that this failure mode raises an alert
   • Change feature flag practice so that a flag cannot be removed while it is still being evaluated
   • Keep a routing service instance out of traffic if it never loaded state from the metadata service

   View case
2. IL-0212 Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queries

   Slack2022

   Symptoms

   Just after 6 a.m. Pacific Time on 22 February 2022, user tickets about problems connecting to Slack, problems reported by internal users, and pages and alerts to several engineering teams all arrived at almost the same moment. Many users could not connect: the Slack client could not finish booting, the step that fetches channel lists, preferences and recent conversations, and showed an error page.

   Database overloadHigh error rateHigh latency

   Trigger

   A percentage-based rollout step upgraded the Consul agent on 25% of the fleet, followed by a slow sequential restart of the agents. Each restart on a Memcached node removed it from the Consul service catalog, so Mcrib, the cache-tier control plane, replaced it with an empty spare or flushed it when it rejoined, and the cache hit rate dropped as Slack reached its daily peak traffic.

   Root cause

   Capacity exhaustion

   Component

   DatabaseIn-memory cacheService discovery

   Contributing factor

   Dependence on a warm cacheInefficient queryAutomation amplificationRetry stormChange during peak hours

   Detection

   User tickets about connection problems, reports from internal users and pages and alerts to several engineering teams arrived almost at the same time, just after 6 a.m. Pacific Time.

   Remediation

   • Paused the Consul agent restart operation
   • Throttled client boot requests to reduce load on the database tier
   • Cut the client boot rate limit again after a too-large increase and then raised it in smaller steps
   • Changed the GDM membership query to read from Vitess only the data missing from Memcached
   • Let the query read from replicas as well as Vitess primaries
   • Raised the client boot rate limit slowly back to normal traffic levels as the caches refilled

   Action items

   • Change the process for Consul rollouts so upgrades do not trigger this failure mode again
   • Change the Mcrib control loop so a brief network outage or partition of cache nodes does not set off the same cascading failure
   • Make the GDM membership query read from a table sharded by channel
   • Check the other cache-fronted database queries for high-volume scatter queries
   • Explore longer-term ways to make the caching tier more resilient

   Lessons

   Slack describes performance bottlenecks, a cascading failure, dependence on a warm cache and a change as the trigger. Once the system had passed the tipping point, lower cache hit rates caused high database load that in turn kept the cache from filling, and a system in that state stays there until an outside intervention, a large cut in load or an increase in capacity, changes it. Mcrib contributed not by being wrong but by being faster than the old Consul lock scheme at replacing Memcached nodes, which increased churn in the cache tier. Client retries still added load despite backoff and jitter. A brief network partition of the cache nodes could set off the same failure, and the interaction between Consul and the caching architecture is now more widely understood.

   View case
3. IL-0251 Most of Slack's HAProxy load balancers routed to a stale, shrinking set of webapp backends and served HTTP 503 errors

   Slack2020

   Symptoms

   At 4:45pm Pacific on 12 May 2020, Slack was alerted that it was serving more HTTP 503 errors than normal. A subset of the webapp fleet was under heavy load while the rest of the webapp instances were not.

   High error rateLoad imbalanceStale endpoint routing

   Trigger

   At the end of the US business day the webapp tier began its usual scale-down, and autoscaling terminated the oldest instances first, which were the only backends left in most HAProxy instances' state.

   Root cause

   Software defect

   Component

   HAProxyService discovery

   Contributing factor

   Automation amplificationNo alert on limit

   Detection

   An alert for more HTTP 503 errors than normal. The alerting meant to catch stale HAProxy state was not working.

   Remediation

   • Did a rolling restart of the HAProxy fleet

   Action items

   • Move all ingress load balancing from HAProxy to Envoy Proxy, with endpoint discovery through an xDS control plane

   Lessons

   The alerting for this exact failure had stopped working, and nobody noticed because the HAProxy stack had run for a long time without needing changes, so few engineers worked with its monitoring and alerting. A slot limit that had been ample a few months earlier was reached after pandemic-driven growth and one large autoscaling event.

   View case
4. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour

   Discord2020sev1

   Symptoms

   On 16 March 2020 at 14:19 PDT most of Discord's Elixir services, which handle real-time connections and chat message processing, crashed at once. Discord lost much of its real-time capacity, the failure cascaded into dependent systems, and most users could not connect for about an hour.

   Service outageHigh error rate

   Trigger

   At 14:18 the Google Cloud managed instance group autoscaler added a node to Discord's API cluster, and the TCP connection of its first announce request to etcd was reset after the request headers were sent but before the body.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discovery

   Contributing factor

   Automation amplification

   Detection

   At 14:21 several internal alarms fired on a drop in key metrics, along with anomaly alerts for cluster utilization, and a SEV1 incident was declared.

   Remediation

   • Restarted the lost real-time capacity and repaired the failed nodes
   • Throttled connections to Discord to help services recover
   • Stopped anti-entropy repairs on two ScyllaDB clusters so reconnecting users would not starve them of resources
   • Raised the connection throttle step by step to let more users reconnect

   Action items

   • Make the etcd clients send each announce request in a single TCP packet, so headers and body arrive together or not at all
   • Add error handling that ignores services whose key value is corrupt instead of crash-looping
   • File an upstream bug report with the Go project about net/http not checking the body against Content-Length

   Lessons

   Discord's service discovery code handled many etcd failure modes but did not expect a client to write a key without a value because of a transient network error. Go's net/http accepts a request whose body is shorter than its Content-Length header, which breaks the HTTP 1.1 specification and added a failure case nobody had anticipated.

   View case

By root cause3