Graph
Cases5 cases
1 to 5 of 5
IL-0133A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius2026- Symptoms
On 19 August 2026 the first equipment-level signs in Nebius's us-central1 region appeared around 09:40 UTC: the InfiniBand fabric degraded and GPU over-temperature alerts began to accumulate. Customer reports followed shortly after 10:00. At 10:15 on-call engineers were paged on critical component temperature alerts, and the incident was declared at 10:21.
- Trigger
A storm-related event at the data center building took the building management system offline and shut down the chilled-water cooling loop.
- Root cause
- Component
- Contributing factor
- Detection
No facility-level alert fired, because the storm also disabled the building management system that monitors the facility. Engineers acted on component over-temperature alerts that paged on-call at 10:15 UTC, after InfiniBand degradation, GPU over-temperature alerts and customer reports; the temperature rise from about 09:15 was visible on dashboards but had no alert.
- Remediation
- had the facility operator switch cooling to manual bypass mode and opened rack rear doors to vent heat
- re-energised tripped racks, control-plane racks first
- restored the regional infrastructure control plane by manual intervention
- restarted block-storage host components to clear stale disk attachment state
- raised the VM recovery rate limits by hand for the region-scale recovery
- started the affected Managed Kubernetes control-plane instances by hand
- applied a controlled stop and start to control-plane instances whose data disk had been left unattached
- identified and started 104 customer VMs that the incident had stopped
- Action items
- Complete a joint resilience review with the facility operator covering redundancy of facility-management and monitoring systems, cooling that keeps running when those fail, electrical protection of cooling equipment and on-site response
- Add facility-level environmental alerting on cooling plant state and room temperature trend that pages on-call independently of the building management system
- Set up a documented emergency escalation path with the facility operator, with an on-call contact tree and guaranteed response times
- Define an emergency heat-load reduction procedure with criteria, ownership and mechanics for shutting down or degrading workloads during a cooling loss
- Establish a dependency-ordered region cold-start procedure with clear ownership for each service layer
- Run regular region-scale recovery drills that cover VMs, storage attachments and Managed Kubernetes clusters
- Fix the disk-attachment bug that extended the Managed Kubernetes outage and deploy the fix to all production regions
- Improve the recovery policies Compute users can configure, and move all Managed Kubernetes control-plane nodes to the improved policy
- Automate reconciliation of stale storage attachment state after abrupt host failures
- Remove in-region dependencies from monitoring startup and recovery, and extend cross-region monitoring
- Make regional health checks used for global traffic management reflect real service availability
- Lessons
Facility monitoring that runs on the same system it watches goes silent in exactly the failure it should report, and with no alert on the room temperature trend there was no practical window to shed heat load in a controlled way. A cold start of a whole region exercised recovery paths built for isolated failures: single-attempt VM recovery, conservative rate limits and stale disk attachments all needed manual work. The longest part of the outage was restoring customer resources, Managed Kubernetes control planes above all, so Nebius is extending its recovery drills to that layer.
- Symptoms
Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.
- Trigger
OpenSearch ran out of disk space and index writes were blocked.
- Root cause
- Component
- Contributing factor
- Detection
A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.
- Remediation
- restarted all fluent-bit containers (logs flowed for a few minutes, then the error came back)
- increased OpenSearch disk space from 8000 to 12000
- configured fluent-bit to stop sending logs to OpenSearch while the root cause was investigated
- Action items
- Look into index dating issues in OpenSearch and Elasticsearch
- Add high-priority alerts for Elasticsearch and OpenSearch
- Re-introduce OpenSearch into live logging
- Investigate the fluent-bit "failed to flush chunk" error
- Symptoms
At 09:14 on 4 August 2023, users of the Cloud Platform reported in the #ask-cloud-platform Slack channel that Kibana was missing their application logs for long periods.
- Trigger
The newer Fluent Bit version in use had changed its chunk drop strategy; the fix that followed handled memory buffer overflow.
- Root cause
- Component
- Contributing factor
- Detection
Users reported long periods of missing logs in Kibana in the #ask-cloud-platform Slack channel at 09:14; the incident was declared at 10:09.
- Remediation
- configured Fluent Bit to handle memory buffer overflow by writing buffered logs to the file system
- made Fluent Bit flush logs in smaller chunks
- Action items
- Push notifications from the logging clusters to #lower-priority-alerts (#4704)
- Add an integration test that checks logs are being sent to the logging cluster
- Symptoms
At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.
- Trigger
The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.
- Root cause
- Component
- Contributing factor
- Detection
A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.
- Remediation
- drained the monitoring node, moving Prometheus to another monitoring node
- changed the node instance size to r6i.4xlarge
- raised the Prometheus memory limit to 100Gi
- set the Prometheus container limits to 12 CPU cores and 110Gi of memory
- Action items
- Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)
IL-0035Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl2019- Symptoms
The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.
- Trigger
A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.
- Root cause
- Component
- Contributing factor
- Detection
Users reported error pages, slow responses and timeouts from several services; no alert fired.
- Remediation
- terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
- found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
- fixed the ElastAlert problem that caused the memory growth
- Action items
- Define alerts on resource usage such as CPU, memory and disk space
- Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
- Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
- Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
- Set resource limits for ElastAlert through the Helm values
- Configure namespaces with default memory requests and limits
- Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
- Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
- Deploy through a pipeline that includes a smoke test step
- Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics
- Lessons
Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.
These 5 cases involve Container memory limit and Prometheus. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause3
Dependency failure1 case
- Contributing factor
- Organisation
Capacity exhaustion3 cases
- Contributing factor
- Organisation
IL-0042Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceIL-0043Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyIL-0016Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
Configuration change1 case
- Contributing factor
- Organisation