Message broker

Cases, by severity

1. IL-0410 SQS Fair Queues rejected valid messages with InvalidParameterValue, failing about half of AssemblyAI's US async requestsAssemblyAI · 2026

   Root cause Cloud provider outage

2. IL-0572 About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a dayMercari · 2026

   Root cause Breaking change in an upgrade

3. IL-0286 An exercise cleanup destroyed Honeycomb's Kafka brokers, leaving partitions leaderless and ingest crash-loopingHoneycomb · 2025

   Root cause Operator error

4. IL-0390 PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalledPostHog · 2025

   Root cause Capacity exhaustion

5. IL-0214 PagerDuty's Kafka brokers ran out of JVM heap one after another, and the event APIs rejected events with 502 responsesPagerDuty · 2025

   Root cause Faulty deployment

6. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hoursGoogle Cloud · 2025

   Root cause Configuration change

7. IL-0545 Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodesWasabi · 2024

   Root cause unknown

8. IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024

   Root cause Capacity exhaustion

9. IL-0285 Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panicsClever Cloud · 2024

   Root cause Configuration change

10. IL-0535 An internal Kinesis Data Streams cell in US-EAST-1 degraded, raising errors in CloudWatch Logs, S3 events and LambdaAmazon Web Services · 2024

    Root cause Software defect

11. IL-0280 Kafka brokers in many New Relic cells ran out of disk, leaving US data collection and alerting unusable for hoursNew Relic · 2021

    Root cause Operator error

12. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumersCircleCI · 2021

    Root cause Breaking change in an upgrade

13. IL-0384 Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failedAmazon Web Services · 2020

    Root cause Capacity exhaustion

14. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeperAppOptics · 2018

    Root cause Cloud provider outage

15. IL-0275 Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hoursParse.ly · 2015

    Root cause Capacity exhaustion