Cases6 cases

1 to 5 of 6

1. IL-0552 CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UI

   CircleCI2025

   Symptoms

   At about 23:00 UTC on 21 January 2025, an automated alert reported that the CircleCI database instance holding archived data was almost out of free storage space. From 23:50 UTC customers saw slow starts and cancellations of workflows and jobs, and delays and difficulty viewing workflows in the CircleCI UI, although pipelines for a specific project could still be viewed.

   High latencyIntermittent timeoutsLog floodNode disk fullCloud API rate limit exhausted

   Trigger

   A code change to a function in the archival database on 16 January 2025 unintentionally made it write an excessive number of log messages, and the database filled its storage.

   Root cause

   Faulty deployment

   Component

   DatabaseCI system

   Contributing factor

   Change with a delayed effectExcessive blast radiusOverlapping failures

   Detection

   An automated alert at about 23:00 UTC warned that the archival database was almost out of free storage, before customers were affected. The growing workflows queue was not linked to it at first because a separate alert about request volume fired at about the same time.

   Remediation

   • Scaled up the event consumer that processes the workflows queue at 23:44 UTC
   • Promoted a read replica of the archival database to a standalone primary at 23:55 UTC

   Action items

   • Fix the database function so it no longer writes an excessive number of log messages
   • Add a max duration to the workflows retry policy for archiving so it fails earlier than the default timeout
   • Move workflow archival to an event-based model that decouples it from the workflows service
   • Add forecast and anomaly monitoring for databases that alerts on unusual activity before it reaches critical levels

   Lessons

   The storage alert on the archival database did fire, but the database filled much faster than forecast, so there was little time to act before customers were affected. Because the workflows service called the archival service synchronously, a problem in the archival database slowed the whole workflows service.

   View case
2. IL-0312 Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went down

   Gadget2024

   Symptoms

   At 11:25 AM EST on 13 December 2024, Gadget restarted its main PostgreSQL server in place to raise max_connections. Postgres shut down but did not start again, and every request to Gadget applications failed, including frontend requests, GraphQL API requests, webhook processing and syncs.

   Service outageHigh latency

   Trigger

   An emergency in-place restart of the main Postgres server at 11:25 AM EST, decided at 11:00 AM, to raise max_connections, which Postgres cannot change without a full restart.

   Root cause

   Capacity exhaustion

   Component

   DatabaseLinux kernelDatabase proxy and connection pooler

   Contributing factor

   Unverified assumptionUnrepresentative stagingAutomation amplificationExcessive blast radius

   Detection

   The team carrying out the planned restart saw at 11:26 AM EST that Postgres had shut down and did not start again.

   Remediation

   • Restarted the database server with a hardware reset, which cleared the fragmented memory so Postgres could start
   • Provisioned almost 4x the compute after the autoscalers had scaled it down during the outage

   Action items

   • Use only the replica-failover approach for disruptive database operations such as restarts
   • Run a more aggressive capacity planning process before Black Friday / Cyber Monday, preparing for a 6x increase in load and proving it with load tests well in advance
   • Announce planned downtime ahead of time on the status page and community Discord, and make the operating procedures tell planned maintenance apart from service disruptions
   • Finish the database sharding project so apps can use different databases and configuration changes can go through canary and blue/green setups
   • Finish building a zero-downtime database failover system
   • Set much higher scaling floors on the compute autoscalers

   Lessons

   Gadget names three deeper factors. Choosing the in-place restart put the database into a state the team had not seen before and was not confident in. Capacity planning for Black Friday / Cyber Monday assumed a doubling of traffic instead of the almost 5x that came, so settings such as max_connections stopped fitting and had to be changed in an emergency instead of in a maintenance window practised in staging. And there was no way to test database configuration changes in a production-like environment: integration tests run against a synthetic Postgres database that never goes through the months of production load behind the memory fragmentation or the memory usage issues.

   View case
3. IL-0456 Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQL

   Inflab2022

   Symptoms

   During a 100% discount event that made 125 courses free, Inflearn received 5 to 20 times its normal traffic, and on four days, 3, 4, 7 and 17 January 2022, the service became unusable around 9 a.m. and 10 p.m. There were six outages in all.

   Service outageDatabase overloadHigh latency

   Trigger

   A 100% discount event on 125 courses, from 3 to 17 January and 18 to 24 January 2022, went viral and brought 5 to 20 times the usual traffic.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Inefficient queryMonitoring blind spotExcessive blast radius

   Detection

   The team followed the outages through hourly transaction counts, RDS metrics, monitoring and slow query alerts.

   Remediation

   • Added an index to the interested-companies query on the course page
   • Changed how often the cache data stored in the database was reset and regenerated
   • Moved the read queries that loaded the writer DB to the reader DBs
   • Scaled the database to 8 r5.8xlarge instances, which bought only a few tens of minutes
   • Rewrote slow course queries or added indexes so they use indexes, and switched off features whose queries could not be fixed
   • Removed the large course description column from the select * course queries
   • Split the IDs of the IN clause into batches of 20 run with Promise.all
   • Upgraded PostgreSQL from 10.16 to 11.13
   • Set a 5-second query timeout
   • Raised max connections
   • Ran VACUUM by hand on the course table
   • Moved the large course description column into a separate table

   Action items

   • Change the structure in which one table's failure spreads to the whole service
   • Improve the table and cache structures that are hard to tune
   • Make the SQL code easier to follow
   • Improve code that gives no confidence to change, such as code without types or test code
   • Build a stable batch environment
   • Keep a monolith of reasonable size and separate the domains through which failures spread most

   Lessons

   Because every database bottleneck appeared as slow queries on the reader DBs, watching only the readers hid the root cause on the writer. In PostgreSQL, frequent updates to a table with large columns produce large dead tuples in bulk and slow every query on that table. Before the second event the team could not run a meaningful performance test because the first week showed that almost any page could fail under load. The event exposed high sensitivity to failure in every area, one table's failure spreading to the whole service, hard-to-tune table and cache structures, unintuitive SQL code, code without types or tests, no stable batch environment, and many hidden failure points not yet found.

   View case
4. IL-0328 A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"

   CircleCI2021

   Symptoms

   From about 18:50 to 19:43 UTC on Monday, November 8, 2021, CircleCI customer jobs on all executors were blocked in a "not running" state. Errors and alerts appeared right after a deploy that changed a field type in the database of the job distribution service.

   Service outageHigh latencyScale-up failureCloud API rate limit exhausted

   Trigger

   At 18:50 UTC a deploy changed the type of a field in the PostgreSQL database of the job distribution service, in a way that was not backwards compatible.

   Root cause

   Faulty deployment

   Component

   DatabaseNode pool

   Contributing factor

   Ineffective rollbackExcessive blast radiusGap in automated tests

   Detection

   Errors and alerts related to the distribution failure appeared immediately after the 18:50 UTC deploy.

   Remediation

   • Rolled back the field type change, which did not clear the errors
   • Wrote, built and manually deployed a change that made the distribution system ignore the field
   • Manually scaled the primary compute cluster for the expected thundering herd of work
   • Monitored until processing was normal and resolved the incident

   Action items

   • Fix the gaps found in the testing strategy
   • Change how the distributor reacts to unexpected data so that this kind of deploy should not affect all jobs
   • Make recovery faster when unforeseen failures occur

   View case
5. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read

   Google Cloud2017

   Symptoms

   From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

   Service outageDatabase overloadHigh latencyHigh error rate

   Trigger

   A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase

   Contributing factor

   Ineffective rollbackExcessive blast radius

   Detection

   Google received an automated alert at 12:34 US/Pacific, one minute after the start.

   Remediation

   • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
   • Ramped Memcache traffic back up gradually while the error rate fell

   Action items

   • Reduce the number of readers of the global configuration to avoid contention during writes
   • Regionalise the configuration to limit the blast radius of similar failures

   View case

By root cause3