Cases2 cases

1 to 2 of 2

1. IL-0251 Most of Slack's HAProxy load balancers routed to a stale, shrinking set of webapp backends and served HTTP 503 errors

   Slack2020

   Symptoms

   At 4:45pm Pacific on 12 May 2020, Slack was alerted that it was serving more HTTP 503 errors than normal. A subset of the webapp fleet was under heavy load while the rest of the webapp instances were not.

   High error rateLoad imbalanceStale endpoint routing

   Trigger

   At the end of the US business day the webapp tier began its usual scale-down, and autoscaling terminated the oldest instances first, which were the only backends left in most HAProxy instances' state.

   Root cause

   Software defect

   Component

   HAProxyService discovery

   Contributing factor

   Automation amplificationNo alert on limit

   Detection

   An alert for more HTTP 503 errors than normal. The alerting meant to catch stale HAProxy state was not working.

   Remediation

   • Did a rolling restart of the HAProxy fleet

   Action items

   • Move all ingress load balancing from HAProxy to Envoy Proxy, with endpoint discovery through an xDS control plane

   Lessons

   The alerting for this exact failure had stopped working, and nobody noticed because the HAProxy stack had run for a long time without needing changes, so few engineers worked with its monitoring and alerting. A slot limit that had been ample a few months earlier was reached after pandemic-driven growth and one large autoscaling event.

   View case
2. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours

   Mailchimp2019

   Symptoms

   From 05:35 UTC on 4 February 2019 Mandrill, Mailchimp's transactional email service, sent only 80% of queued emails, because one of its PostgreSQL shards, shard4, stopped accepting writes. When Mandrill engineers opened an investigation at 11:58 UTC, they saw a sharply elevated load on shard4.

   Writes rejectedThroughput degradationLoad imbalanceNode disk full

   Trigger

   At 05:35 UTC on 4 February the transaction ID counter of shard4, the busiest Mandrill PostgreSQL shard, reached its upper limit and PostgreSQL shut the database down to protect against wraparound.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Ignored early signalNo alert on limitSignal buried in noiseRetry stormMonitoring blind spot

   Detection

   The first exception was logged overnight but was buried by unrelated exceptions in the centralized error reporting. Mandrill engineers opened an investigation at 11:58 UTC and, while looking for the source of the rising load on shard4, found the wraparound at 12:56 UTC.

   Remediation

   • Put shard4 into standalone mode and started VACUUM at 14:16 UTC on 4 February
   • Stopped and restarted the vacuum with settings that made it run faster
   • Started a dump and restore of the database without the Search and Url tables while the vacuum ran
   • Replaced storage volumes on the Mandrill app servers whose disks were filling with queued jobs and logs
   • Moved the job-queue tables from shard4 to other live shards
   • Truncated the Search and Url tables on the locked database at 19:12 UTC on 5 February, after which the vacuum finished in about an hour
   • Started sending from the backlog at 21:36 UTC
   • Added alerting for XID wraparound by 22:40 UTC

   Lessons

   Mailchimp named the factors that slowed its response: knowledge of and access to Mandrill systems and monitoring rested with a few people, Mandrill logging needed a locally writable disk, logs of malformed API requests were hard to separate from real application errors, job health was hard to see, and the disk space problem made partial deployments for triage risky. What helped was trained incident commanders with clear roles (incident commanders in 3-hour rotations, scribes, technical liaisons and points of contact between teams), periodic summaries with a list of recent and open decisions, and taking time in a video meeting to reassess, which led to the truncation. Running in the cloud let the team expand storage quickly, and the team was lucky that the largest table holding up the vacuum held data it could delete.

   View case

By root cause2