Cases7 cases

1 to 5 of 7

1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts

   MUSINSA2026

   Symptoms

   The on-call engineer for a high-traffic service was alerted that requests were failing in large batches every 30 seconds.

   High error rateIntermittent timeoutsConnection pool exhaustion

   Trigger

   DEBUG level logging was on in production when traffic surged, raising log volume to about 100 times normal while the service logged synchronously.

   Root cause

   Configuration change

   Component

   Application logging libraryContainer runtimeTelemetry agentDatabase

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   The on-call engineer was alerted that requests were failing in large batches every 30 seconds.

   Action items

   • Replace synchronous logging with TraceAwareAsyncAppender, which captures trace_id and span_id on the calling thread and drops logs instead of blocking when the queue is full (neverBlock=true)
   • Keep the amount of log output at a sustainable level, since an asynchronous queue still fills under sustained overload

   Lessons

   Removing AsyncAppender to keep trace IDs also removed the protection that kept request threads away from stdout back-pressure. A company-wide standard applies the same side effect to dozens of services, so check what a standard rests on and what working behaviour it takes away. ThreadLocal context does not cross into an asynchronous worker, but capturing it on the calling thread before handing the event off keeps both the IDs and the asynchronous output. Under extreme load, dropping log lines is preferable to stopping request handling.

   View case
2. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table

   IONOS Cloud2026

   Symptoms

   On 3 March 2026, during a scheduled maintenance to replace a faulty switch in a cluster in IONOS Cloud's TXL data center, monitoring alerts fired and the hosts in that cluster lost network connectivity. Customers saw intermittent network degradation or outages, and the Data Center Designer (DCD) web frontend and the AI Model Hub became unreachable.

   Network unreachableLink flappingService outage

   Trigger

   During a scheduled maintenance to replace a faulty switch, Topology 2 of 2 in the affected cluster was disabled as planned, and a link flap on a switch in Topology 1 then flooded the InfiniBand control plane with alerts and started a fabric resweep.

   Root cause

   Capacity exhaustion

   Component

   InfiniBand fabricNetwork control plane (SDN)BGP routingNIC, cable and switch port linkModel hubDatabase

   Contributing factor

   Automation amplificationExcessive blast radius

   Detection

   IONOS began investigating monitoring alerts in TXL; the first status update was posted at 11:36 UTC.

   Remediation

   • Switched from the affected control plane to its standby
   • Rebooted the locked-up gateways
   • Recovered Topology 1 from the scheduled maintenance
   • Paused provisioning in the cluster that had a backlog of queued jobs
   • Re-enabled provisioning once network connectivity improved and watched job execution
   • Re-established connectivity for the systems that support the DCD

   Action items

   • Adjusted the control-plane configuration to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table
   • Reduced the number of multicast group memberships to relieve the forwarding table
   • Set up continuous monitoring of forwarding table use and control-plane memory with alerts before capacity limits are reached
   • Move the control plane to dedicated high-performance servers, separate from data-plane traffic
   • Audit IPv6 configurations and IPoIB driver settings
   • Modernise the interconnect fabric, including switches, gateways and drivers
   • Hardened IAM maintenance protocols so that critical identity services keep working when the primary Cloud APIs are temporarily unavailable
   • Move IAM to a dedicated platform without a dependency on the standard managed Kubernetes clusters
   • Introduce a dedicated provisioning provider for DBaaS and Managed Kubernetes
   • Add automated circuit breakers to the provisioning engine against overload and queue buildup from self-healing services
   • Improve visibility and control over job execution and prioritisation in the provisioning queue
   • Change the provisioning queue architecture to decouple services and reduce spillover

   Lessons

   IONOS identified the linking of services through the shared provisioning queue as a key reason the degradation lasted as long as it did. Alongside making each service more robust, it is decoupling the provisioning queue and addressing dependencies between services to reduce the blast radius of future incidents.

   View case
3. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck

   Scaleway2024

   Symptoms

   On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

   Network unreachablePacket lossNode NotReady

   Trigger

   At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API

   Contributing factor

   Excessive blast radiusAutomation amplification

   Detection

   Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

   Remediation

   • Fixed both VPC route reflectors by manual action

   Action items

   • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
   • Run different software versions on the route reflectors so a single bug cannot stop several instances
   • Investigate the software issue that left the route reflectors in a stale state

   View case
4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitDatabase

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
5. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast

   Microsoft Azure2023

   Symptoms

   From about 10:30 UTC on 30 August 2023, customers had trouble accessing or using Azure, Microsoft 365 and Power Platform services in Australia East. The affected Azure services included Azure Kubernetes Service (AKS) along with storage, SQL Database, Cosmos DB, virtual machines and Azure Resource Manager.

   Control plane unresponsiveHigh error rateService outage

   Trigger

   At 08:41 UTC a lightning strike on electrical infrastructure about 18 miles away caused a utility voltage sag that shut down the cooling chillers of datacenters in one Australia East availability zone; 13 chillers failed to restart automatically and five could not be restarted, so two data halls overheated and their infrastructure was shut down.

   Root cause

   Cloud provider outage

   Component

   AKS control planeDatabase

   Contributing factor

   Single point of failureExcessive blast radius

   Detection

   Cosmos DB impact was detected by monitoring at 10:47 UTC, first automated communications went to Azure Service Health at 10:48 UTC, the public Azure Status page was updated at 11:30 UTC, and all subscriptions in Australia East were sent portal communications at 11:36 UTC.

   Remediation

   • shut down the remaining networking, compute and storage in the two affected data halls so the chilled water loop could cool down
   • restarted the five chillers manually at 12:12 UTC
   • restored power to all hardware by 15:10 UTC and brought infrastructure back in phases
   • restored AKS operations through the recovery of the SQL Database operation queue, with no other mitigation

   Action items

   • AKS: convert all operation queue SQL Database databases to be zone redundant (estimated September 2023)
   • AKS: replace all cross-region SQL Database queue usage with zone-redundant Service Bus queues (estimated September 2023)
   • Change the control timing logic of the Chiller Management System so chillers do not fail to restart after voltage fluctuations
   • Change the emergency procedure for manual restarts of simultaneous chiller failures from an adjacent sequence to a load-based sequence per data hall
   • Increase technician staffing at the datacenter as a temporary measure for manual chiller restarts
   • Storage: optimise the large-scale recovery process and update troubleshooting guides (estimated December 2023)
   • Azure Service Fabric: improve reliability of SQL Database tenant ring recovery (estimated December 2023)
   • SQL Database: review the trigger criteria of auto-failover groups (estimated October 2023)
   • SQL Database: upgrade internal tooling to enable mass migration of databases (estimated December 2023)
   • Cosmos DB: optimise Service Managed Failover for single-region-write accounts (estimated November 2023)
   • ARM: complete the migration of its storage layer to the zonally redundant architecture (estimated December 2023)
   • Incident management: harden readiness, process and playbook for power-down scenarios (estimated October 2023)

   Lessons

   A control plane spread across availability zones still fails with a zone when one of its dependencies, here the SQL Database operation queue, is not zone redundant. When a second region uses the same database, the failure crosses into that region as well. Workloads already running were not expected to be affected; what stopped was everything that needs the managed service's control plane, such as scaling and cluster changes.

   View case

By root cause4