Signal buried in noise

Cases

1. IL-0515 Downloads from files.pythonhosted.org failed with intermittent 502 and 503 errors for two weeks, breaking PyPI installsPython Software Foundation · 2026

   Root cause Dependency failure

2. IL-0286 An exercise cleanup destroyed Honeycomb's Kafka brokers, leaving partitions leaderless and ingest crash-loopingHoneycomb · 2025

   Root cause Operator error

3. IL-0214 PagerDuty's Kafka brokers ran out of JVM heap one after another, and the event APIs rejected events with 502 responsesPagerDuty · 2025

   Root cause Faulty deployment

4. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database clusterWikimedia Foundation · 2025

   Root cause Faulty deployment

5. IL-0280 Kafka brokers in many New Relic cells ran out of disk, leaving US data collection and alerting unusable for hoursNew Relic · 2021

   Root cause Operator error

6. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat37signals · 2020

   Root cause Dependency failure

7. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addressesGandi · 2020

   Root cause Software defect

8. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster serviceDFDS · 2020

   Root cause Configuration change

9. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hoursMailchimp · 2019

   Root cause Capacity exhaustion

10. IL-0399 Applications on GOV.UK PaaS in AWS eu-west-1 failed DNS lookups intermittently for about two hours overnightGovernment Digital Service · 2016

    Root cause Cloud provider outage

11. IL-0337 EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicasAmazon Web Services · 2011

    Root cause Operator error