Incident Library
Applied filters SymptomHigh error rateFactorAutomation amplificationOrganisationHoneycombClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023sev1
    Symptoms

    Standard alerting paged Honeycomb's on-call engineers about the performance and stability of event ingestion, and the automated load-shedding mechanisms tripped and sent further notifications. The status page described a higher than normal error rate and response time on the ingestion service.

    High error rateHigh latencyContainer crash loopService outageDatabase overload
    Trigger

    An abnormally large number of queries arrived from many separate connections within a few minutes.

    Root cause
    Capacity exhaustion
    Component
    Application databaseIn-memory cache
    Contributing factor
    Retry stormAutomation amplification
    Detection

    Standard alerting notified the on-call engineers about ingestion performance and stability, and the automated load-shedding mechanisms added notifications when they tripped. The first status post, about alerts on a higher than normal error rate and response time, went up at 18:28 PST (02:28 UTC), 13 minutes after the 02:15 UTC start.

    Remediation
    • Scaled the ingest fleet up manually and aggressively
    • Cut off most ingest for a few minutes so the ingest hosts could come back with enough capacity
    • Temporarily lowered the rate limits for the most aggressive traffic sources until their traffic subsided
    Action items
    • Keep the ingest fleet scaled up to prevent a repeat while longer-term fixes are worked on
    • Decide on the projects that will address these failure paths
    Lessons

    Load shedding and auto-scaling were in place, yet a burst that arrived faster than the cache could spread its writes still took the ingest hosts down. During recovery, Kubernetes' crash loop back-off worked against the operators: the hosts that had failed stayed offline while the fleet needed every host. Clients that retry whole OTel batches after a partial drop can multiply traffic many times over, so per-source rate limits were what finally stabilised ingest.

    View case
Check your cluster for this

This case involves In-memory cache and Application database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
In-memory cacheApplication database
Contributing factor
Automation amplificationRetry storm
Organisation
Honeycomb
  1. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023