Cases4 cases

1 to 4 of 4

1. IL-0290 GitLab.com's Redis Sidekiq cluster ran out of memory and lost quorum, causing 500 errors and a 12 million job backlog

   GitLab2026sev1

   Symptoms

   From 17:12 UTC on 12 May 2026, error rates rose at the same time across GitLab.com's web, Git, API and CI runner services. Users got frequent 500 errors, failed or stuck CI pipelines, authentication failures and unreliable Git access, and a Prometheus Alertmanager alert opened the incident at 17:15.

   High error rateMemory pressureThroughput degradation

   Trigger

   Earlier on 12 May, as a mitigation for a previous incident, INC-10084, a Sidekiq queueing apdex SLO violation on shard catchall-b, jobs of AuditEvents::AuditEventStreamingWorker were deferred. The deferred audit event jobs then piled up in the Redis Sidekiq cluster.

   Root cause

   Capacity exhaustion

   Component

   In-memory cache

   Contributing factor

   Change with a delayed effectPages not deliveredLimited debug access

   Detection

   A Prometheus Alertmanager alert reported the incident at 17:15:10 UTC, three minutes after impact began. A follow-up asks why the redis-sidekiq memory alert did not page, and the review says some signals did not reach the right on-call people.

   Remediation

   • Restarted and force-reset the two unresponsive Redis Sidekiq nodes
   • Stopped each Redis Sidekiq VM in turn and resized it to c2-standard-16, doubling memory and CPU
   • Re-enabled AuditEventStreamingWorker instead of deferring it

   Action items

   • Investigate why the redis-sidekiq memory alert did not page
   • Establish the root cause, failure mode and remediation status of the WebHookWorker poison pill
   • Give mitigations that disable safety limits an explicit owner and a deadline for re-evaluating them
   • Stop AuditEvents::AuditEventStreamingWorker from causing recurring incidents
   • Create a runbook for AuditEventStreamingWorker
   • Address PG connection pool saturation

   Lessons

   Broad degradation across web, Git, authentication, CI runners, Sidekiq and the internal API made it hard to settle on one cause and explain it clearly. The customer symptom of pipelines stuck in created was hard to measure in real time, and the dashboards that would have answered it were not obvious to on-call responders. Some alerts did not reach the right on-call people, and hosts that hung on login slowed diagnosis and made changes riskier. Redis-sidekiq under memory saturation forces choices between availability and data loss that are hard to make during an incident and call for guardrails and design changes.

   View case
2. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat

   CyberAgent2026

   Symptoms

   An alert fired one day because an application's error rate had suddenly risen. The error log of the Aurora MySQL RDS instance for that time showed Aurora killing connections and declining queries to avoid running out of memory. The killed queries also made application latency worse.

   High error rateMemory pressure

   Trigger

   An application release added a batch job run every minute by a Kubernetes CronJob. Each run's short-lived pod connected to the database from a new pod IP.

   Root cause

   Capacity exhaustion

   Component

   DatabaseJobs and CronJobs

   Contributing factor

   Unreviewed defaultsChange with a delayed effect

   Detection

   An alert fired on the application's error rate.

   Remediation

   • Set performance_schema to manual management and capped two related Performance Schema parameters, unlimited by default, at 100 in the parameter group, then restarted the RDS instance

   Lessons

   When system alerts fire but the database's load metrics look calm, Performance Schema growth is worth suspecting. The two parameters that were capped are unlimited by default, so connections from ever-changing source addresses keep adding to the records. Capping them has a cost: once the cap is exceeded, per-host figures in Performance Insights are missing. Putting RDS Proxy between the application and Aurora MySQL to keep source hosts stable is another possible measure.

   View case
3. IL-0214 PagerDuty's Kafka brokers ran out of JVM heap one after another, and the event APIs rejected events with 502 responses

   PagerDuty2025

   Symptoms

   When a Kafka cluster of PagerDuty's US Service Regions began failing at 03:53 UTC on 28 August 2025, the first alerts pointed to one failing broker. More brokers then ran out of memory, and PagerDuty's event APIs began rejecting incoming events with a response status code of 502 while notifications, webhooks, chat integrations and the REST API degraded.

   Memory pressureHigh error rateThroughput degradation

   Trigger

   A new feature that recorded API key usage in a Kafka topic was being rolled out by configuration: 1% on 21 August, 5% and then 25% on 27 August, and 75% on 28 August. A logical error in the feature created a new Kafka producer for every API request instead of reusing one producer.

   Root cause

   Faulty deployment

   Component

   Message broker

   Contributing factor

   Monitoring blind spotSignal buried in noiseChange with a delayed effectExcessive blast radius

   Detection

   Alerts pointed to one failing Kafka broker at the start of the first incident, and alerts fired again at 16:38 UTC when Kafka performance degraded for the second time. Critical alerts about core API errors were buried among webhook pages.

   Remediation

   • Expanded the Kafka cluster and removed the first failing broker
   • Doubled the Kafka broker heap size and applied it with a rolling restart across the cluster
   • Added status page updates by hand through backup procedures when automated publishing failed
   • Recovered Kafka-dependent services, some by manual intervention, and backfilled lost data with scripts
   • In the second incident, doubled the heap again with another rolling restart
   • Rolled back the API usage feature with a pull request that stopped it from publishing Kafka messages

   Action items

   • Remove the dependency that blocked status page automation and simplify and republish the manual runbook
   • Run live drills regularly so status updates go out even when parts of the platform are degraded
   • Expand JVM and Kafka monitoring of heap, garbage collection and producer and consumer health
   • Strengthen service dependency mapping so cascading failures are easier to trace
   • Collect customer impact metrics into incident workflows automatically
   • Add stricter change management guardrails, with readiness checklists and slower ramp-up windows for major rollouts
   • Improve recovery mechanisms in the streaming systems
   • Automate communication workflows such as status page updates, internal and external syncs and reminders
   • Test major incident workflows monthly with chaos engineering, such as status page failover
   • Define clearer incident severity levels and roles, including dedicated communications liaisons

   Lessons

   PagerDuty found its message processing more intricate than it had thought: some services had to be restarted to pick up changes, some API activity produced far more messages than expected, and it was hard to tell which parts of the system put heavy load on Kafka. Because producers and JVM heap were barely observed, responders could not see the first failure until several downstream systems were affected, and no one on the call held a model of the whole system. Enabling the feature through configuration rather than a separate deployment was meant as a safety step but made the outage harder to link to the change. Runbooks and backfill scripts were imperfect but gave responders a head start, and the second incident was mitigated in about 50 minutes by repeating the first incident's steps.

   View case
4. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable

   Enigmo2020

   Symptoms

   Two or three days after Enigmo created EKS managed node groups and moved its pods there from the old node groups, the cluster suddenly became unstable: pods restarted again and again or could not stop and stayed around indefinitely, node CPU and memory load rose high, and the nodes became unusable from the cluster.

   Pods stuck in TerminatingSustained high node CPUMemory pressureNode NotReadyUpgrade stall

   Trigger

   Enigmo created EKS managed node groups that used an AMI containing the bug and moved its pods onto them, as preparation for upgrading EKS from 1.15 to 1.16.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imageNode pool

   Contributing factor

   Ineffective rollbackChange with a delayed effect

   Remediation

   • Changed the managed node groups' AMI version to the fixed release
   • Ran the managed node group update through CloudFormation
   • Watched the pods across all namespaces during the update
   • Force-deleted every pod that stayed in Terminating with kubectl delete --grace-period=0 --force

   Action items

   • Check the latest releases of the cluster's modules quarterly or monthly and upgrade them regularly

   Lessons

   As long as a team uses Amazon EKS or Kubernetes, it has to keep track of upgrades of the cluster and of modules such as ingress controllers, because a release with breaking changes can be announced without warning after a long stable period. Checking releases and upgrading on a regular schedule keeps the effect of each upgrade on the service small.

   View case

By root cause3