IL-0206 GitLab · 2022-04-25kubernetesgcp
Every GitLab.com pod on GKE runs a Consul agent, which the Rails application uses to discover which PostgreSQL databases are healthy, up-to-date replicas. Under certain circumstances the agent's memory demand spikes, and with a pod memory limit of 500MB the agent was killed again and again. Without a working Consul agent, the Rails pods (web, api, git and websockets) kept failing to discover replicas and fell back to sending all their queries to the last known primary database. The primary saturated, and Sidekiq processing and the whole site slowed down intermittently on 2022-04-25 and 2022-04-26.
During the first incident the team worked from a different hypothesis: that a pgbouncer pool reduction made on 2022-04-21 had saturated connections. The plan was to raise the per-user connection limit for the gitlab database user and the pgbouncer backend pools for the main database, and to consider rolling back the CI feature flag from 50%. Saturation and slowness dropped and the incident was marked mitigated, but the elevated error rate in the early hours of 2022-04-26 turned out to be related, and further investigation tied both events to the Consul OOM kills.
As a temporary fix, the pod memory limits were raised from 500MB to 1GB so the Consul agents keep running through a spike. The team planned to watch the agents with the new limit and to find out whether the spikes are expected Consul behaviour or come from GitLab's configuration.
On 2022-04-25 the Sidekiq Apdex (the share of background jobs that finish within a target time) dropped, and an engineer declared an incident in Slack at 15:13 UTC. Users saw intermittent slowness across GitLab.com, and asynchronous work such as pipeline creation and import and export took unusually long.
The first suspect was a pgbouncer change made on 2022-04-21 that had reduced connection pool sizes. At 16:23 the working hypothesis was that this reduction had saturated database connections; the team also looked at other recent deploys and at a feature flag rollout at 50% for CI phase 4. The Sidekiq queue had been in a bad state at 15:33 but had recovered by 16:03, and one engineer pointed out that the saturation metric was slightly incorrect, so saturation might be lower than it looked.
The incident was marked mitigated at 17:08 because saturation and site slowness had dropped, and changes were applied at 17:26 (the timeline does not list them). At 01:21 UTC on 2022-04-26 monitoring detected an elevated error rate, first declared as a separate incident and later tied to the same cause. Further investigation found OOM kills on the GKE nodes: under certain circumstances the Consul agent in each pod had a spike in memory demand, and when the memory was not available it was killed rapidly and repeatedly.
Root cause Capacity exhaustion
An engineer declared the incident in Slack at 15:13 UTC after the Sidekiq Apdex dropped; the second event was found when monitoring detected an elevated error rate at 01:21 UTC on 2022-04-26.
When the local Consul agent that Rails uses for database service discovery keeps dying, Rails does not fail; it sends every query to the last known primary. What engineers saw was database saturation and slow background jobs, which pointed the investigation first at a recent pgbouncer pool change. The memory limit of the Consul agent in every pod therefore decided the load on the primary database.
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit, Application database
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Container memory limit
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Application database
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Container memory limit
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
IL-0027same root cause · shared Container memory limit
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Application database
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Container memory limit
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Container memory limit
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Application database
IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Application database
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit