IL-0333 Buildkite · 2016-08-22databaseaws
A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com down
Buildkite had grown on AWS Activate credits by adding hardware. About two weeks before the incident the team found the credits were about to run out and set out to cut the bill quickly. The largest cost was an m4.10xlarge Multi-AZ RDS PostgreSQL database, which was downgraded during a weekend maintenance window to an r3.2xlarge Multi-AZ instance. Because the new instance had less memory, connection pooling was moved from pgbouncer on each host to one centralised pgbouncer. Neither change was load tested. The database looked fine over the weekend and into what the team took for the start of the US peak, and the team went to bed.
At about 16:00 UTC the database CPU maxed out. Buildkite runs three ELBs with Auto Scaling Groups, for the dashboard, the REST API and the Agent API, and the health check behind them tested both SQL databases, Redis and Memcached. Health checks returned HTTP 500, the dashboard ELB removed its servers, and buildkite.com went offline at 17:21 UTC. Replacement servers deployed themselves from a baked AMI and looked up the current revision from buildkite.com itself; when that failed they ran the AMI's old code, whose health check referenced a decommissioned database, so they failed and were terminated in a loop. The PagerDuty alerts from New Relic, Pingdom, Datadog, Calibre and CloudWatch did not wake anyone: the on-call engineer's "Immediately phone me" notification rule had dropped out of his settings, so only silent push notifications arrived, and the other team members, reached by escalation, had their phones on silent.
The team woke at 21:00 UTC. An AWS IAM issue blocked the database upgrade and the launch of a fixed AMI in the AWS Console, so at 22:28 UTC servers were taken from the Agent API and added to the dashboard ELB, bringing buildkite.com back with slow responses. When IAM recovered, the EC2 request limit, used up by the server churn, forced the team to wait. The team switched the Auto Scaling Groups to EC2 health checks and locked their desired counts, then used the aws-cli to upgrade the database to an instance type that could handle the load and roll out servers whose self-deploy worked, and went back to pgbouncer on each host. Buildkite fully recovered at 23:10 UTC.
- Severity
- –
- Scope
- buildkite.com dashboard down: no login, no build logs, no documentation; builds kept running and pull request statuses kept updating; slow responses and dropped requests from 22:28 to 23:10 UTC
- Users affected
- –
- Region
- –
- Time to detect
- 3h 39m
- Time to resolve
- 5h 49m
Symptoms
On 22 August 2016 at 17:21 UTC buildkite.com went offline: nobody could log in, view build logs or read the documentation, although builds kept running and GitHub and Bitbucket pull request statuses kept updating. The Buildkite team in Melbourne was asleep and woke at 21:00 UTC, almost 4 hours later, to emails, tweets and Slack messages saying Buildkite was down.
At about 16:00 UTC, during US peak load, the RDS PostgreSQL database that had been downgraded over the weekend could no longer keep up and its CPU maxed out. The health check endpoint behind each Elastic Load Balancer ran SELECT 1 against both SQL databases and pinged Redis and Memcached, so it began returning HTTP 500 errors saying it could not connect to the database. The ELB in front of the dashboard had the lowest failed health check threshold and marked its servers OutOfService; the ELBs for the REST API and the Agent API had problems but kept their servers.
The Auto Scaling Group launched replacement servers, but every one of them failed its health check and was terminated, and this launch-and-terminate cycle went on for hours. New servers were built from an AMI with an old version of the code baked in and found the current Git revision by requesting https://buildkite.com/_secret/version, which could not answer while buildkite.com was down. The baked-in code's health check still referenced Job::Log::Chunk::Shard::One, a database that had been decommissioned.
Recovery was slowed by AWS. An IAM problem at AWS stopped the team from upgrading the database in the AWS Console, and when the team tried to launch a fixed AMI for testing, the IAM role selector did not work and the AWS Console kept showing "Internal server error". Once IAM recovered, new EC2 servers could not be launched because the launch-and-terminate cycle had used up the EC2 request limit. After a partial recovery at 22:28 UTC, responses stayed very slow and some requests were dropped.
Symptom Service outageDatabase overloadProbe failuresCloud API rate limit exhausted
Root cause
Root cause Capacity exhaustion
- Trigger
- Over a weekend maintenance window, Buildkite downgraded its m4.10xlarge Multi-AZ RDS PostgreSQL database to an r3.2xlarge Multi-AZ instance to cut its AWS bill, and moved to a centralised pgbouncer to pool connections. The first US weekday peak after the change pushed the new instance's CPU to its maximum at about 16:00 UTC.
Detection
New Relic, Pingdom, Datadog, Calibre and CloudWatch raised PagerDuty alerts, but the on-call phone call never came and only silent push notifications arrived. The team noticed at 21:00 UTC when they woke to emails, tweets and Slack messages from users.
Remediation
- Moved servers from the Agent API to the dashboard ELB to bring buildkite.com back online
- Switched the Auto Scaling Groups to EC2 health checks and locked in desired counts
- Upgraded the database to an instance type that could handle the load, using the aws-cli
- Built a new AMI that did not depend on buildkite.com/_secret/version and rolled out servers that could self-deploy
- Removed the centralised pgbouncer and went back to pgbouncer on each host
Action items
- Plan load testing after significant infrastructure changes without affecting other users of the system
- Make the ELB health check return OK when the app answers, without checking database connectivity
- Consider a separate health check endpoint for Pingdom that checks the database and other dependencies
- Store the latest deployed revision as a file on S3 for newly launched servers to self-deploy from
- Test by hand that PagerDuty calls reach every on-call team member
- Investigate regular company-wide tests of everyone's paging setup
- Ask PagerDuty about calling members when they go on call or requiring an immediate phone rule
Lessons
Buildkite listed six lessons. Watch AWS credits, because billing reports show $0 while credits are spent, so the real cost is hidden and big changes end up rushed. Load test after significant infrastructure changes. Think about what a health check should do and what happens when every server behind an ELB fails it. Do not make new servers look up the deployed revision from the service they are replacing. Make sure every on-call member has the right PagerDuty and phone settings before going on call. When servers churn in AWS, stop health checks and autoscaling, for example with the Auto Scaling feature to suspend and resume processes, so the EC2 request limit is not used up.
Sources
References
Related cases
IL-0379InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026same root cause · shared Database, Liveness and readiness probes
IL-0450Oracle RAC commits behind order APIs kept waiting over 0.1 s on log file sync while average host CPU had headroomToss Securities · 2026same root cause · shared Database
IL-0553CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutesCircleCI · 2026same root cause · shared Database
IL-0309Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hoursPreset · 2026same root cause · shared Database
IL-0435Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flatCyberAgent · 2026same root cause · shared Database
IL-0497Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passedButtondown · 2026same root cause · shared Database
IL-0288ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026same root cause · shared Database
IL-0350Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud · 2026same root cause · shared Database
IL-0305Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026same root cause · shared Database
IL-0390PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalledPostHog · 2025same root cause · shared Database
IL-058129CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week saleMUSINSA · 2025same root cause · shared Liveness and readiness probes
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Database
IL-0287One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operationsMongoDB · 2025same root cause · shared Database
IL-0557Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hoursStorj · 2025same root cause · shared Database
IL-0558New PubNub instance types in FRA hit connectivity limits, and Pub/Sub, History and Presence saw intermittent errorsPubNub · 2025same root cause · shared Cloud load balancer
IL-0551Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a dayAtlassian · 2025same root cause · shared Database
IL-0298Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disksWikimedia Foundation · 2025same root cause · shared Database
IL-0312Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went downGadget · 2024same root cause · shared Database
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared Database
IL-0447An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baselineevery · 2024same root cause · shared Database
IL-0375MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a databaseMongoDB · 2024same root cause · shared Database
IL-0110A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Database
IL-0550Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pendingAtlassian · 2024same root cause · shared Database
IL-0270Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutesDiscord · 2023same root cause · shared Database
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Database
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared Database
IL-0326A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hoursMailPace · 2023same root cause · shared Database
IL-0546Buildkite jobs could not go to newly connected agents once agent IDs passed a 32-bit integer foreign key's limitBuildkite · 2023same root cause · shared Database
IL-0500UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up2i2c · 2022same root cause · shared Database
IL-0301Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice37signals · 2022same root cause · shared Database
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Database
IL-0302Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022GitHub · 2022same root cause · shared Database
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared Database
IL-0456Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQLInflab · 2022same root cause · shared Database
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021same root cause · shared Database
IL-0578Static content behind an HAProxy load balancer in an OpenStack Octavia amphora was delayed by tens of secondsGMO Pepabo · 2020same root cause · shared Cloud load balancer
IL-0496Ably's Cassandra hit 100% CPU in every region and queries timed out globally, and the load returned on two more eveningsAbly · 2020same root cause · shared Database
IL-0443A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019same root cause · shared Database
IL-0037Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Database
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared Database
IL-0211A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hoursMailchimp · 2019same root cause · shared Database
IL-0064CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Database
IL-0300Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,64737signals · 2018same root cause · shared Database
IL-0334Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went downInstapaper · 2017same root cause · shared Database
IL-0282DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata serviceAmazon Web Services · 2015same root cause · shared Database
IL-0383PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming eventsPagerDuty · 2014 · sev1same root cause · shared Database
IL-0320Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0333
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0333/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.