Incident Library
Applied filters Root causeCapacity exhaustionSymptomWrites rejectedClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

7 to 8 of 8

  1. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions
    cloud.gov2016
    Symptoms

    At 7:12pm Eastern Time on 5 July 2016, logs.cloud.gov, the cloud.gov log search service, sent users into a login redirect loop, so they could not log in to the service.

    Service outageWrites rejected
    Root cause
    Capacity exhaustion
    Component
    In-memory cacheElasticsearch or OpenSearch
    Contributing factor
    Overlapping failuresMonitoring blind spot
    Remediation
    • Moved Redis on the logsearch service to a much larger memory instance so it could keep logs while Elasticsearch was down
    • Corrected the Elasticsearch memory limits and the /tmp noexec hardening setting and redeployed the cluster
    • Redeployed the Kibana instance and verified the logs once the cluster deployment was complete
    Action items
    • Add a failure alert that checks whether logs are coming through
    • Make Redis auto-scaling or highly available so that failures in more fragile downstream components do not cause data loss
    • Extend the incident response plan to start with events that are not security-specific
    • Create a chaos bot for the logs.cloud.gov service
    View case
  2. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed
    Postmark2011
    Symptoms

    At about 8:15am EDT Postmark's primary MySQL server ran out of disk space, and email sending, the API and the web interface were delayed. Disk space alerts fired, but the space had grown much faster than usual and MySQL was already struggling when the alerts were noticed. MySQL kept running while it failed to write some data to disk, so MySQL-MMM, which normally detects problems and fails over to a healthy server, never triggered a failover; the team spent about 30 minutes trying to free logs and disk space before failing over by hand.

    Node disk fullWrites rejectedFailover failure
    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase failover manager
    Contributing factor
    –
    Detection

    Per-server monitoring raised disk space alerts, but the disk filled so quickly that MySQL was already struggling by the time the alerts were noticed.

    Remediation
    • Tried to free up logs and disk space on the primary server
    • Failed over manually to the second MySQL server
    • Recovered the jobs that had failed
    • Sent the emails that each server had queued locally while the database was unavailable
    Action items
    • Upgrade the MySQL cluster with a focus on availability and performance
    • Make the alerts more sensitive to catch problems sooner
    Lessons

    Looking back, Postmark said it should have failed over to the second server right away instead of first making sure replication was healthy.

    View case
Check your cluster for this

These 8 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion8 cases

Component
Database6In-memory cache2PostgreSQL vacuum1Database backups and snapshots1Database proxy and connection pooler1Database replication1Elasticsearch or OpenSearch1Database failover manager1
Contributing factor
Monitoring blind spot5Inefficient query2Signal buried in noise1Unverified assumption1Untested backup1Excessive blast radius1Change during peak hours1Ignored early signal1Unknown limit1Limited debug access1Harmful manual action1No alert on limit1No capacity planning1No runbook1No headroom1Overlapping failures1Retry storm1Unsafe failure behaviour1
Organisation
37signals1cloud.gov1GitHub1Hello Inc.1Instapaper1Mailchimp1MailPace1Postmark1
  1. IL-0432 AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limit
    Hello Inc.2026
  2. IL-0326 A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hours
    MailPace2023
  3. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022
    GitHub2022
  4. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours
    Mailchimp2019
  5. IL-0300 Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,647
    37signals2018
  6. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017
  7. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions
    cloud.gov2016
  8. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed
    Postmark2011