Cases2 cases

1 to 2 of 2

1. IL-0372 Heroku apps could not connect to about 1% of hobby-tier Postgres databases after the databases' host names changed

   Heroku2021

   Symptoms

   From 00:00 UTC on 25 February 2021, Heroku apps attached to about 1% of hobby-tier Postgres databases could not connect to their databases. Heroku first posted that it was investigating connectivity issues with Hobby tier databases at 03:39 UTC.

   Stale endpoint routingHigh error rate

   Trigger

   An upgrade process in an upstream system wrongly changed the host names of several shared database systems.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseService discovery

   Contributing factor

   –

   Remediation

   • Halted the upstream upgrade process
   • Updated the relevant systems with the new host names and published a new release on the affected apps
   • Gave customers a workaround of rotating database credentials with heroku pg:credentials:rotate

   Action items

   • Improve monitoring to detect similar incidents
   • Provide tooling and documentation that shorten recovery from a similar incident

   View case
2. IL-0251 Most of Slack's HAProxy load balancers routed to a stale, shrinking set of webapp backends and served HTTP 503 errors

   Slack2020

   Symptoms

   At 4:45pm Pacific on 12 May 2020, Slack was alerted that it was serving more HTTP 503 errors than normal. A subset of the webapp fleet was under heavy load while the rest of the webapp instances were not.

   High error rateLoad imbalanceStale endpoint routing

   Trigger

   At the end of the US business day the webapp tier began its usual scale-down, and autoscaling terminated the oldest instances first, which were the only backends left in most HAProxy instances' state.

   Root cause

   Software defect

   Component

   HAProxyService discovery

   Contributing factor

   Automation amplificationNo alert on limit

   Detection

   An alert for more HTTP 503 errors than normal. The alerting meant to catch stale HAProxy state was not working.

   Remediation

   • Did a rolling restart of the HAProxy fleet

   Action items

   • Move all ingress load balancing from HAProxy to Envoy Proxy, with endpoint discovery through an xDS control plane

   Lessons

   The alerting for this exact failure had stopped working, and nobody noticed because the HAProxy stack had run for a long time without needing changes, so few engineers worked with its monitoring and alerting. A slot limit that had been ample a few months earlier was reached after pandemic-driven growth and one large autoscaling event.

   View case

By root cause2