Cases10 cases

1 to 5 of 10

1. IL-0497 Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passed

   Buttondown2026

   Symptoms

   On 31 March 2026 Buttondown was down twice, for about seven and about thirteen minutes. New requests could not get a database connection and failed, although the database itself was healthy: queries were fast, CPU and memory were fine and replication lag was nominal.

   Connection pool exhaustionService outage

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningMonitoring blind spotCircular dependency

   Detection

   User reports and the team's own manual observation, not automated monitoring. The health check endpoint kept returning 200 for most requests, so no alert fired.

   Remediation

   • Killed active queries to free database connections in the first outage
   • Restarted the database to force-kill the running queries in the second outage

   Action items

   • Reserve a small number of database connections for administrative access
   • Monitor and alert on the database connection count as a percentage of the configured maximum
   • Make the health check endpoint run a lightweight query so connection exhaustion shows in the health checker and the load balancer
   • Add database connection limits to the quarterly capacity review
   • Document and script a fallback that force-recycles workers to release connections without needing a database connection

   Lessons

   Buttondown notes that the tool used to recover from the first outage needed the same resource that was exhausted, which locked the team out during the second outage, and that a health check which cannot detect connection exhaustion, the most common failure mode, gives little protection. A configuration value set long ago drifted out of line with traffic because nobody reviewed it.

   View case
2. IL-0284 Workers KV failed 90.22% of requests in a third-party storage outage, breaking Access, WARP and many Cloudflare products

   Cloudflare2025

   Symptoms

   At 17:52 UTC on 12 June 2025, Cloudflare's WARP team saw registrations of new devices failing and declared an incident. Minutes later the Cloudflare Access team got an alert on a rapid rise in error rates, and Service Level Objectives for several other services fell below target. All of them shared one cause: Workers KV was unavailable.

   High error rateService outageCredentials unavailable

   Trigger

   Part of the storage infrastructure behind Workers KV is backed by a third-party cloud provider, and that provider had an outage. Workers KV could then no longer serve cold reads or writes from its central data store.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Single point of failureExcessive blast radiusCircular dependencyRetry storm

   Detection

   The WARP team saw registrations of new devices failing at 17:52 UTC and declared an incident. At 18:05 the Access team received an alert on a rapid increase in error rates and Service Level Objective alerts fired across several teams; at 18:06 the incidents were combined once Workers KV unavailability was identified as the shared cause.

   Remediation

   • Activated the Challenges and Turnstile kill switches so visitors were not blocked
   • Degraded Gateway rules that referenced identity or device posture state to remove the Workers KV dependency
   • Force-dropped Access identity and device posture requests to shed load on Workers KV
   • Worked on a Workers KV release against an alternative backing datastore in case the storage stayed down
   • Restored Access and Device Posture calls to Workers KV once the third-party service was back

   Action items

   • Avoid singular dependencies on storage infrastructure that Cloudflare does not own
   • Improve the ability to recover critical services, including Access, Gateway and WARP
   • Add redundancy to Workers KV's storage infrastructure and remove the dependency on any single provider
   • Make each affected product resilient to the loss of any single point of failure, third-party dependencies included
   • Build tooling to re-enable namespaces progressively during storage incidents, so services come back without overloading Cloudflare's own infrastructure

   Lessons

   Cloudflare holds itself responsible for its choice of dependencies and how it architects around them, even when a vendor failure was the trigger. Building many products on Workers KV normally makes them more robust than separate storage services would, but here the cascading failure of Workers KV made the outage worse and widened its blast radius.

   View case
3. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case
4. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours

   Square2023

   Symptoms

   From 1:54 PM ET on 7 September 2023, Square products and services were unavailable, and at 2:02 PM ET engineers were notified of infrastructure issues and began investigating DNS. Square Online websites stayed up, but their customers could not make payments.

   DNS resolution failuresService outage

   Trigger

   At 11:04 AM ET a host-based firewall rule change was deployed to enable region communication, which increased the size of the firewall rules on nodes. Combined with a DNS service upgrade, it put unexpected load on the internal DNS servers. The timeline also lists a DNS zone change at 1:56 PM ET.

   Root cause

   Configuration change

   Component

   Authoritative DNS servers and recordsNode iptables rules

   Contributing factor

   No staged rolloutCache-masked failureCircular dependency

   Detection

   Engineers were notified of infrastructure issues at 2:02 PM ET, eight minutes after services became unavailable.

   Remediation

   • Shed network traffic to the DNS servers and started bringing up new DNS servers by hand
   • Deployed networking changes by hand to re-enable the authorization and access services
   • Deployed the networking changes by hand to all data centers
   • Rebuilt the DNS servers
   • Built a new configuration from the reverted firewall ruleset and pushed it to the DNS hosts

   Action items

   • Move the DNS infrastructure onto isolated infrastructure
   • Add monitoring and optimizations for critical networking infrastructure
   • Optimize dependencies between the deploy and platform infrastructure where feasible
   • Expand support for Offline Mode and improve communication about when it is available

   View case
5. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case

By root cause7