Incident Library
Applied filters SymptomAutoscaler misbehaviourOrganisationSentryClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024
    Sentry2024
    Symptoms

    On 21 October 2024 Sentry could not ingest data in US SaaS from 9:42 AM to 10:06 AM PT, and the status page reported elevated errors for ingestion. The same failure had also happened on 4 and 16 October.

    High error rateService outageAutoscaler misbehaviour
    Trigger

    In each of the three incidents an ingestion region saw an increase in backlogged traffic; the trigger was different each time.

    Root cause
    Capacity exhaustion
    Component
    CDN and edge networkAutomated capacity manager
    Contributing factor
    Automation amplificationExcessive blast radius
    Action items
    • Create more, smaller edge regions to reduce the chance of cascading failures, adding one region each in South America, North America and Europe
    • Change the autoscaling rules to use stronger signals than CPU load
    • Scale down more slowly so that iowait-based backlogs do not trigger scale-down events
    • Improve the metrics the edge produces to refine the autoscaling rules and their sensitivity
    View case
Check your cluster for this

This case involves Automated capacity manager and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Automated capacity managerCDN and edge network
Contributing factor
Automation amplificationExcessive blast radius
Organisation
Sentry
  1. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024
    Sentry2024