Cases24 cases

1 to 5 of 24

1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned

   Nebius2026

   Symptoms

   On 19 August 2026 the first equipment-level signs in Nebius's us-central1 region appeared around 09:40 UTC: the InfiniBand fabric degraded and GPU over-temperature alerts began to accumulate. Customer reports followed shortly after 10:00. At 10:15 on-call engineers were paged on critical component temperature alerts, and the incident was declared at 10:21.

   Control plane unresponsiveService outageMonitoring outage

   Trigger

   A storm-related event at the data center building took the building management system offline and shut down the chilled-water cooling loop.

   Root cause

   Dependency failure

   Component

   Data center power and coolingCloud block storageetcdInfiniBand fabric

   Contributing factor

   Single point of failureMonitoring blind spotNo alert on limitNo runbookOverlapping failures

   Detection

   No facility-level alert fired, because the storm also disabled the building management system that monitors the facility. Engineers acted on component over-temperature alerts that paged on-call at 10:15 UTC, after InfiniBand degradation, GPU over-temperature alerts and customer reports; the temperature rise from about 09:15 was visible on dashboards but had no alert.

   Remediation

   • had the facility operator switch cooling to manual bypass mode and opened rack rear doors to vent heat
   • re-energised tripped racks, control-plane racks first
   • restored the regional infrastructure control plane by manual intervention
   • restarted block-storage host components to clear stale disk attachment state
   • raised the VM recovery rate limits by hand for the region-scale recovery
   • started the affected Managed Kubernetes control-plane instances by hand
   • applied a controlled stop and start to control-plane instances whose data disk had been left unattached
   • identified and started 104 customer VMs that the incident had stopped

   Action items

   • Complete a joint resilience review with the facility operator covering redundancy of facility-management and monitoring systems, cooling that keeps running when those fail, electrical protection of cooling equipment and on-site response
   • Add facility-level environmental alerting on cooling plant state and room temperature trend that pages on-call independently of the building management system
   • Set up a documented emergency escalation path with the facility operator, with an on-call contact tree and guaranteed response times
   • Define an emergency heat-load reduction procedure with criteria, ownership and mechanics for shutting down or degrading workloads during a cooling loss
   • Establish a dependency-ordered region cold-start procedure with clear ownership for each service layer
   • Run regular region-scale recovery drills that cover VMs, storage attachments and Managed Kubernetes clusters
   • Fix the disk-attachment bug that extended the Managed Kubernetes outage and deploy the fix to all production regions
   • Improve the recovery policies Compute users can configure, and move all Managed Kubernetes control-plane nodes to the improved policy
   • Automate reconciliation of stale storage attachment state after abrupt host failures
   • Remove in-region dependencies from monitoring startup and recovery, and extend cross-region monitoring
   • Make regional health checks used for global traffic management reflect real service availability

   Lessons

   Facility monitoring that runs on the same system it watches goes silent in exactly the failure it should report, and with no alert on the room temperature trend there was no practical window to shed heat load in a controlled way. A cold start of a whole region exercised recovery paths built for isolated failures: single-attempt VM recovery, conservative rate limits and stale disk attachments all needed manual work. The longest part of the outage was restoring customer resources, Managed Kubernetes control planes above all, so Nebius is extending its recovery drills to that layer.

   View case
2. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed

   ANDPAD2026

   Symptoms

   At about 17:00 JST on Thursday 16 July 2026, Datadog alerted, and at almost the same time 5xx errors began to rise on the Amazon CloudFront distributions that used VPC Origins connections. Builder Work, ANDPAD's job and career service for the construction industry, became almost entirely unreachable.

   Service outageHigh error rate

   Trigger

   The Amazon CloudFront VPC Origins feature failed; AWS Health later reported a capacity constraint in an internal shared subsystem that manages connections for VPC Origins.

   Root cause

   Cloud provider outage

   Component

   CDN and edge networkCloud load balancerIngress controllerWeb application firewall rules

   Contributing factor

   Single point of failure

   Detection

   Datadog raised an alert, and at about the same time 5xx errors were seen rising on the CloudFront distributions that used VPC Origins.

   Remediation

   • Decided to route traffic temporarily through an internet-facing ALB without CloudFront
   • Recreated the internet-facing Ingress, attached it to the Argo Rollouts Rollout resource and promoted it
   • Created a regional WAF for the ALB with the security team
   • Created a new IngressClass with the ACM certificate, the WAF and access log settings
   • Added Ingresses for normal traffic, the admin screens and redirects, which restored routing

   Action items

   • Return to the CloudFront and VPC Origins layout while keeping a way to switch to internet-facing in code
   • Keep the internet-facing ALB, limited to approved CIDRs through spec.inboundCIDRs in IngressClassParams, so commenting out securityGroupInboundCidrs in values.yaml opens it to 0.0.0.0/0
   • Point the Terraform-managed Route53 A record back to CloudFront and keep the ALB alias commented out in code, so uncommenting and applying switches DNS to the ALB

   Lessons

   VPC Origins keeps the ALB off the internet and simplifies the traffic path, but it makes the service depend on an AWS-managed shared subsystem, and when that fails the service stops even though CloudFront is healthy. ANDPAD kept VPC Origins and instead prepared the escape route ahead of time, accepting the cost of an ALB that always runs, because recreating the Ingress and attaching it to the Rollout had taken so long during this incident.

   View case
3. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks

   AdGuard VPN2025

   Symptoms

   The day began with an alert that several instances of the Redis cluster were running out of disk space. That part was fixed quickly. The outage came an hour later: right after the Redis StatefulSet was deleted and recreated at 8:10 UTC, Kubernetes restarted every Redis pod within about five minutes, and problems started to appear.

   Service outageOOMKilledProbe failuresHigh latency

   Trigger

   To make Git match the volumes that had been resized by hand, the Redis StatefulSet was deleted with orphan (non-cascading) deletion and recreated with the new storage template; Kubernetes reconciled the new spec by restarting the entire Redis cluster.

   Root cause

   Configuration change

   Component

   Persistent volumes and stateful setsIn-memory cacheContainer memory limit

   Contributing factor

   No health probesUnverified assumptionSingle point of failureRetry stormNo headroom

   Detection

   An alert about Redis instances running out of disk space started the work; the outage itself showed up as backend errors about Redis connections right after the StatefulSet was recreated.

   Remediation

   • made the Redis masters forget obsolete node identities, reset the affected replicas and rejoined them, then waited for replication to catch up
   • identified the component that was actually running out of memory, scaled the backend service and raised memory limits for that component
   • temporarily relaxed probes so backend instances took traffic immediately instead of flapping between ready and not ready
   • reverted the temporary workarounds at 10:00 UTC

   Action items

   • Add startup probes so a Redis pod is not marked ready until its dataset is loaded
   • Build an own control plane for Redis that handles restarted pods stuck to an old master address
   • Reduce dependence on the single Redis cluster: move some important data to other stores, such as a separate Redis cluster per server, and use Kafka for some backend components

   Lessons

   The disk resize was not the problem; the cleanup afterwards was. An immutable field pushes operators toward deleting and recreating the controller, and that can set off a full reconcile. Startup and readiness probes are what slow a rolling restart down to a safe pace; without them a distributed store restarts in the most fragile way. Recovery plans must expect a cluster to heal slowly, and dependent services need to degrade gracefully while it does.

   View case
4. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours

   Wealthy2025sev1

   Symptoms

   Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

   Scale-up failureContainer crash loopService outageDNS resolution failures

   Trigger

   Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

   Root cause

   Breaking change in an upgrade

   Component

   GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets

   Contributing factor

   Deferred upgradesSingle point of failureIneffective rollbackNo headroom

   Remediation

   • Created a new GKE cluster at v1.33, the version of the forced upgrade
   • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
   • Updated the cert-manager configuration for Kubernetes 1.33
   • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
   • Decommissioned the broken cluster

   Action items

   • Configure all clusters to use the GKE stable release channel
   • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
   • Review spot instance usage for critical workloads and add on-demand fallback pools
   • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
   • Document a Kubernetes, Istio and cert-manager compatibility matrix
   • Create an automated upgrade testing pipeline
   • Reserve static IPs for quick migrations
   • Pin versions of Istio, cert-manager, CSI and DNS addons
   • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines

   Lessons

   GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

   View case
5. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution

   Mercari2025sev1

   Symptoms

   During peak hours, when Mercari's GKE clusters handle very high request rates, the platform team saw kube-dns reaching its limits. DNS query rates were high enough to overwhelm the kube-dns service.

   DNS resolution failuresIntermittent timeoutsService outage

   Trigger

   Peak-hour traffic on the GKE clusters produced more DNS queries than the few kube-dns pods could answer.

   Root cause

   Capacity exhaustion

   Component

   kube-dns

   Contributing factor

   No cachingSingle point of failure

   Detection

   During peak hours the team saw frequent NXDOMAIN and truncated-response errors and recurring "failed to refresh DNS cache" errors on the clusters, which showed that kube-dns had reached its limits.

   Remediation

   • deployed NodeLocal DNSCache on the Laboratory GKE cluster with a specific nodeAffinity to measure its effect
   • rolled NodeLocal DNSCache out gradually to the Dev and Prod environments by labelling node pools so its pods could be deployed there

   Lessons

   When every pod in a busy cluster resolves names through a few central kube-dns pods, DNS becomes a shared bottleneck, and a DNS failure quickly turns into higher latency, timeouts and cascading outages across microservices. A cache on each node takes most of that load off kube-dns and keeps resolution local to the node. Rolling the cache out first to an internal test cluster with nodeAffinity, and then node pool by node pool through labels, kept the blast radius of the change small.

   View case

By root cause10