IL-0555 Sentry · 2024-10-21networking

Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024

Sentry had three incidents, on 4, 16 and 21 October 2024, with the same root causes and different triggers. Each began with a backlog in one ingestion region of the edge; traffic shifted from that region to others, which also backlogged and rejected traffic, so the failure cascaded and affected customers in the US region. The most severe part of each incident came from a scaling flaw in the edge design. The edge autoscaled on the CPU load of the ingestion servers, which was too slow to detect impact; a slowdown that showed as iowait instead of CPU caused no scaling at all, and in some cases autoscaling took the wrong action entirely. On 21 October data could not be ingested in US SaaS from 9:42 to 10:06 AM PT.

Sentry had already seen the edge scaling problem in prior incidents and says these three are related to incidents earlier in October. Work on it had been under way for weeks: Sentry added more, smaller edge regions, changed the autoscaling rules to use stronger signals than CPU load, and slowed scale-down so that iowait backlogs would not trigger it. Sentry reports that these changes have already brought a large increase in stability and in the speed of recovery.

Severity: –
Scope: Sentry US event ingestion (errors, transactions, attachments, profiles, replays, spans, cron monitoring): backlogs and rejected traffic; no data ingested in US SaaS from 9:42 to 10:06 AM PT on 21 October 2024
Users affected: –
Region: US
Time to detect: –
Time to resolve: 24m

Symptoms

On 21 October 2024 Sentry could not ingest data in US SaaS from 9:42 AM to 10:06 AM PT, and the status page reported elevated errors for ingestion. The same failure had also happened on 4 and 16 October.

In each incident one ingestion region of Sentry's edge built up a backlog of traffic. Traffic moved off that region to other regions, which then backlogged too and rejected traffic, and the impact reached customers in the US region. Each incident lasted only a few minutes, but cleanup and impact visible to customers went on longer.

Autoscaling of the edge did not absorb the backlog: it reacted too slowly or performed the wrong scaling action. The edge scaled on the CPU load of the ingestion servers, which detected the impact too late, and when the slowdown showed up as iowait instead of CPU the edge did not scale at all; backlogs caused by iowait could even set off scale-down events.

Symptom High error rate Service outage Autoscaler misbehaviour

Root cause

Root cause Capacity exhaustion

Component CDN and edge network Automated capacity manager

Contributing factor Automation amplification Excessive blast radius

Trigger: In each of the three incidents an ingestion region saw an increase in backlogged traffic; the trigger was different each time.

Detection

–

Remediation

–

Action items

Create more, smaller edge regions to reduce the chance of cascading failures, adding one region each in South America, North America and Europe
Change the autoscaling rules to use stronger signals than CPU load
Scale down more slowly so that iowait-based backlogs do not trigger scale-down events
Improve the metrics the edge produces to refine the autoscaling rules and their sensitivity

Lessons

–

Sources

https://status.sentry.io/incidents/tft6bj7h9yg8 · official-status · archived copy

Related cases

IL-0353 Fly.io edge proxies in US west coast regions were overloaded and flapping, with lazy loader delays of secondsFly.io · 2026
same root cause · shared CDN and edge network
IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hoursCloudflare · 2025
same root cause · shared CDN and edge network

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CDN and edge network, Automated capacity manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0555
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0555/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.