Cases9 cases

1 to 5 of 9

1. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale

   Buildkite2026

   Symptoms

   From 20:01 UTC on 28 May 2026, Buildkite build and job notifications (GitHub commit status, email, Slack and webhook notifications) reached a subset of customers late. Internal queue latency monitoring showed the delay as queues grew while notification-processing workers stayed underprovisioned.

   High latencyAutoscaler misbehaviour

   Trigger

   The Prometheus service used by Buildkite's EKS autoscaling path ran out of storage, so some EKS-based notification workers could not autoscale while their queues were growing.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   No alert on limitPartial rolloutIneffective rollbackUnverified assumptionHarmful manual action

   Detection

   Internal queue latency monitoring detected the rising notification delivery latency.

   Remediation

   • Moved the affected notification-processing workloads from EKS back to the ECS-based infrastructure
   • Adjusted scaling configuration and worker counts by hand
   • Completed the rollback and scaled the remaining affected ECS services
   • Increased Prometheus storage capacity and recorded the change in infrastructure as code

   Action items

   • Alert before Prometheus storage exhaustion can affect autoscaling
   • Fix GitHub notification retry behaviour for a class of errors that could cause repeated retries
   • Make the EKS-to-ECS rollback verify destination capacity, autoscaling configuration and traffic movement
   • Review the CPU, memory and storage allocations of EKS control-plane dependencies such as KEDA and Karpenter
   • Move customer-critical workloads to EKS more gradually, with clearer settling periods
   • Improve customer-level monitoring of notification delivery latency to detect customer-impacting regressions earlier
   • Review which notification types sit on customers' scheduling or CI hot path and need tighter latency, separate queueing or specific alerting
   • Update the status page earlier when notification latency is likely to affect customer workflows
   • Describe the customer-visible impact in status page updates, not only the affected internal service
   • Improve internal escalation for customers who report critical CI impact

   Lessons

   Buildkite said recovery took longer than expected because the rollback path from EKS to ECS did not fully handle this failure, and that its public status page did not show the customer-visible impact as quickly or clearly as it should have, since notification latency affects customers differently depending on how their CI and deployment workflows use notifications.

   View case
2. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout

   IONOS Cloud2026

   Symptoms

   On 4 March 2026 customers of the IONOS Container Registry got 504 Gateway Timeout errors when pushing or pulling container images, and deployments that relied on the registry were blocked. The status page first reported an increased error count on the registry at 09:23 UTC, saying customers might be unable to pull images.

   Image pull failuresHigh error rate

   Trigger

   A migration combined with a temporary capacity constraint placed two critical control plane components on the same proxy instance, despite anti-affinity rules; the shared proxy then reached its concurrent connection limit.

   Root cause

   Capacity exhaustion

   Component

   Container registry and its credentialsManaged Kubernetes control plane and provider APIPod scheduling and placement

   Contributing factor

   No headroomSingle point of failureNo alert on limit

   Remediation

   • Rolled back a component of the Kubernetes control plane to an earlier version
   • Provisioned additional proxy capacity
   • Moved the affected control plane components onto separate proxy instances

   Action items

   • Redesign registry-to-API connectivity so each node uses its own local proxy
   • Adjust alert thresholds to warn before proxy connection use approaches capacity
   • Deploy more infrastructure clusters and redistribute registries so no cluster exceeds safe capacity, with automation that keeps balancing load

   View case
3. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries

   PostHog2025

   Symptoms

   On 21 October 2025 at 21:45 UTC a change to timeout handling was deployed to the feature flags service, and two minutes later automated monitoring reported higher error rates. The change was rolled back by 21:49, but error rates stayed high: about 38% of flag evaluation requests in the US datacenter returned errors. At 22:30 Redis metrics showed memory exhaustion on ElastiCache, at 22:35 Postgres connections spiked, and at 22:45 the team found a massive data transfer from Postgres into Redis. The main PostHog application was affected as well, because it shares the Redis instance with the flags service.

   Sustained high node CPUHigh error rateHigh latencyContainer crash loopDatabase overload

   Trigger

   On 21 October a deployment meant to reduce timeout errors (PR #39821); on 24 October the deployment of IP-based rate limiting (PR #40074) that saw all traffic as coming from the load balancer's IP; on 28 October an unrelated routine deployment that rolled the feature flag pods in the US region.

   Root cause

   Capacity exhaustion

   Component

   Node poolPod scheduling and placementIn-memory cacheDatabase

   Contributing factor

   No alert on limitRetry stormIneffective rollbackSingle point of failureNo staged rolloutUnrepresentative staging

   Detection

   On 21 October automated monitoring detected higher error rates two minutes after the deploy. On 24 October no alert existed for 429 errors, and the team learned of the incident from customer reports that began at 18:10 UTC, a 62-minute detection delay. On 28 October on-call received a high error count alert. On 29 October the incident was reported with higher error rates and latency. No alert existed for CPU pressure, so the CPU saturation behind all three incidents was not seen directly.

   Remediation

   • rolled back the timeout handling change within 2 minutes on 21 October, which did not stop the errors
   • throttled connections and Redis writes on 21 October
   • halted the rollout on 28 October, scaled the US flags pods to zero until Redis recovered, then scaled them back gradually
   • rolled back hardware changes on 30 October at 00:03 UTC, which removed most errors but not the latency
   • increased the pod resource requests and adjusted the Kubernetes fleet size so fewer pods ran on each node
   • added retry limits to Envoy between the load balancer and the service
   • moved database connection timeouts and other critical settings to runtime configuration and raised the timeout values
   • added per-node and per-pod CPU alerts at 80% sustained for 5 minutes that page on-call

   Action items

   • Form a strike team from the flags, ingestion and infrastructure teams to review the application and infrastructure for remaining bottlenecks
   • Investigate separating the flags Redis instance from the application Redis instance
   • Before re-enabling ArgoCD sync, align the flags service chart configuration with the values in ArgoCD and define the deployment and Redis strategy
   • Make the feature flags team hard code-owners for flags-related code
   • Build a dashboard of key flag metrics with runbook links and update the feature flag runbooks
   • Add missing alerts on existing service and infrastructure metrics
   • Make readiness checks validate dependencies that degrade under load, for example by pinging the database
   • Rate-limit cache population so it cannot overwhelm Redis
   • Monitor connection pools with automatic throttling near exhaustion and limit concurrent connections
   • Build a production-scale load testing framework
   • Build progressive rollout infrastructure to limit blast radius
   • Document Postgres-to-Redis data flows and write runbooks for data transfer storms

   Lessons

   Nodes that are small relative to pod requests let the scheduler pack many pods on each node, and the resulting CPU saturation showed up three different ways: excessive parallelism, connection pool exhaustion and slow queries. Treating each symptom separately let the problem return; right-sizing the pods and nodes fixed all three. CPU must be alerted on from the start. Unbounded Envoy retries and unbounded cache population amplified the overload, and a Redis instance shared with the main application spread a flags problem to the main application. A rollback did not end the incident, because the data transfer and resource exhaustion it had started kept running.

   View case
4. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space

   UK Ministry of Justice2024

   Symptoms

   Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.

   Monitoring outage

   Trigger

   OpenSearch ran out of disk space and index writes were blocked.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentElasticsearch or OpenSearch

   Contributing factor

   No alert on limit

   Detection

   A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.

   Remediation

   • restarted all fluent-bit containers (logs flowed for a few minutes, then the error came back)
   • increased OpenSearch disk space from 8000 to 12000
   • configured fluent-bit to stop sending logs to OpenSearch while the root cause was investigated

   Action items

   • Look into index dating issues in OpenSearch and Elasticsearch
   • Add high-priority alerts for Elasticsearch and OpenSearch
   • Re-introduce OpenSearch into live logging
   • Investigate the fluent-bit "failed to flush chunk" error

   View case
5. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy

   UK Ministry of Justice2023

   Symptoms

   At 09:14 on 4 August 2023, users of the Cloud Platform reported in the #ask-cloud-platform Slack channel that Kibana was missing their application logs for long periods.

   Monitoring outage

   Trigger

   The newer Fluent Bit version in use had changed its chunk drop strategy; the fix that followed handled memory buffer overflow.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agent

   Contributing factor

   No alert on limitMonitoring blind spot

   Detection

   Users reported long periods of missing logs in Kibana in the #ask-cloud-platform Slack channel at 09:14; the incident was declared at 10:09.

   Remediation

   • configured Fluent Bit to handle memory buffer overflow by writing buffered logs to the file system
   • made Fluent Bit flush logs in smaller chunks

   Action items

   • Push notifications from the logging clusters to #lower-priority-alerts (#4704)
   • Add an integration test that checks logs are being sent to the logging cluster

   View case

By root cause1