Cases3 cases

1 to 3 of 3

1. IL-0349 IONOS Private Cloud in Berlin lost management access, storage and VM internet routing after a VLAN change

   IONOS Cloud2026

   Symptoms

   On 15 January 2026, network services for VMware-based Private Clouds in IONOS Cloud's Berlin location stopped working: customers lost management VPN access and traffic to and from their virtual machines. Inside IONOS, several management interfaces in the Berlin region could no longer be reached.

   Network unreachableStorage disconnected or stale mounts

   Trigger

   During a planned maintenance window to add a new VLAN to the management devices, a human error in the command replaced the existing VLANs on the trunk between the core and management devices instead of adding to them, so only the new VLAN stayed active.

   Root cause

   Operator error

   Component

   Network switchTransit gateway or network interconnect

   Contributing factor

   Excessive blast radiusNo safeguard on destructive change

   Remediation

   • Restarted the affected storage services in the final stage of recovery, which caused a brief second outage

   Action items

   • Audit the management device architecture to segment it better (target Q3 2026)
   • Evaluate and implement ways to prevent unintended VLAN changes, such as improved configuration safeguards or automation scripts (target Q1 2026)

   View case
2. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on them

   Microsoft Azure2024

   Symptoms

   From 21:40 UTC on 18 July 2024, virtual machines in the Azure Central US region lost access to their disks, and the services running on those VMs failed. Azure SQL Database and SQL Managed Instance showed unhealthy nodes and login failures, Cosmos DB requests failed, and service management operations failed across many Azure services.

   Storage disconnected or stale mountsService outageHigh error rateFailover failureStale endpoint routing

   Trigger

   Routine changes to the VM host fleet led to a new allow list of VM host address ranges being generated and published to all Storage scale units in Central US. Recent network configuration changes had made some of the infrastructure file servers that hold the source data unreachable from the server generating the list, so the published list was incomplete.

   Root cause

   Configuration change

   Component

   Cloud block storageDatabaseDatabase failover manager

   Contributing factor

   Unsafe failure behaviourNo staged rolloutExcessive blast radiusMonitoring blind spot

   Detection

   Service monitoring detected the drop in VM availability at 22:06 UTC while the allow list update was being published. SQL monitoring detected unhealthy nodes and login failures within one minute of SQL becoming unavailable, and monitoring flagged the availability drop for Cosmos DB, SQL DB and SQL DB Managed Instance at 22:14 UTC.

   Remediation

   • Halted the allow list update workflow worldwide
   • Paused all deployments in the Central US region
   • Paused compute buildout deployments in all regions
   • Updated the configuration of all Storage scale units in Central US, restoring storage availability by 02:55 UTC
   • Started geo-failovers for SQL databases with automatic failover policies
   • Restarted SQL gateway nodes to refresh their caches and route connections to the right nodes
   • Forced completion of SQL failovers that had not completed
   • Failed over the Cosmos DB control plane and active-passive accounts with a write region in Central US
   • Cancelled customer-triggered graceful switch region operations that blocked Cosmos DB failovers
   • Removed locks on Cosmos DB service metadata to force-complete stuck failovers
   • Repaired impacted Cosmos DB partitions for single-region accounts
   • Repaired backend nodes and failed Cosmos DB accounts back to Central US, redoing failback where it failed

   Action items

   • Make the allow list generation workflow detect incomplete source information and halt
   • Alert on storage requests rejected by allow list checks
   • Deploy allow lists one Availability Zone and storage type at a time, over 24 hours
   • Add VM health checks and an automatic stop to the allow list deployment workflow
   • Reevaluate the policy for Microsoft-managed failover of SQL failover groups
   • Fix the Cosmos DB failback workflow that broke some MongoDB drivers for MongoDB API accounts
   • Fix Cosmos DB failback for accounts with private endpoints so it does not need redoing
   • Make the storage data-plane firewall reject invalid allow lists and keep the last-known-good state
   • Improve Azure NetApp Files monitors so impacting events are detected and classified in time
   • Monitor more Azure NetApp Files service metrics to correlate similar events faster
   • Make the SQL Service Fabric cluster location change notifications reliable under load
   • Make SQL geo-failover workflows more robust so they complete
   • Remove the issues that delayed 5% of Cosmos DB failovers
   • Migrate Azure DevOps customer metadata to the right geography
   • Decouple Azure NetApp Files regional reads and writes to keep this class of issue to one region
   • Evaluate caching to reduce Azure NetApp Files reliance on stored utilization data
   • Add automatic per-partition failover for Cosmos DB multi-region active-passive accounts
   • Build the Resilient Ephemeral OS disk improvement so VMs survive storage incidents better

   View case
3. IL-0337 EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicas

   Amazon Web Services2011

   Symptoms

   From 12:47 AM PDT on 21 April 2011, a subset of Amazon EBS volumes in one Availability Zone of the US East Region could no longer serve reads or writes, and EC2 instances that tried to use them got stuck as well. AWS called these "stuck" volumes. EBS API calls across the whole US East Region also returned high error rates and latencies during two periods on the first day.

   Storage disconnected or stale mountsNetwork unreachableControl plane lagHigh error rateFailover failureData loss

   Trigger

   During a routine network change to upgrade the capacity of the primary EBS network in one Availability Zone, traffic was to be shifted off one of the redundant routers. The shift was executed incorrectly and sent the traffic onto the lower-capacity replication network instead of the other primary router, which could not carry it.

   Root cause

   Operator error

   Component

   Cloud block storageDatabase

   Contributing factor

   Automation amplificationRetry stormNo headroomExcessive blast radiusSignal buried in noise

   Remediation

   • Rolled back the incorrect traffic shift, which restored network connectivity
   • Disabled all new Create Volume requests in the affected Availability Zone at 2:40 AM PDT
   • Disabled all communication between the degraded EBS cluster and the EBS control plane from 8:20 AM PDT
   • Stopped EBS servers in the degraded cluster from futilely contacting other servers for free space
   • Changed the EBS control plane so new EBS-backed instance launches stopped failing
   • Moved spare server capacity physically from across the US East Region into the degraded cluster
   • Adjusted the negotiation throttles so the new servers could join without flooding the old ones
   • Built a separate EBS control plane partitioned to the affected zone and finer-grained throttles, then processed the backlog
   • Restored the remaining volumes from S3 snapshots taken early in the event
   • Restored volumes on failed machines that had no snapshot after forensic work
   • Intervened by hand on the RDS multi-AZ instances that did not fail over automatically

   Action items

   • Audit the change process and add automation to prevent the traffic shift mistake
   • Change capacity planning and alarming to keep the extra safety capacity needed for large recovery events
   • Change the retry logic in EBS nodes to back off harder and reconnect to previous replicas instead of searching for new nodes
   • Test and deploy the fix for the race condition that crashed EBS nodes
   • Fix the RDS bug that kept multi-AZ instances from failing over automatically
   • Improve the EBS control plane timeout logic to prevent thread exhaustion when one zone's cluster is slow
   • Make the EBS control plane aware of Availability Zones so it sheds load when over capacity
   • Move more EBS control plane functions into per-cluster services that run in the same zone as the cluster
   • Offer multiple Availability Zones to Amazon VPC customers
   • Provide better tools for building multi-AZ applications that survive the loss of a zone
   • Host free webinars and publish whitepapers on fault-tolerant architecture
   • Build the fine-grained control and throttling used in recovery into the EBS nodes and automate the recovery models
   • Look at changes that keep volumes working while a cluster is degraded, including the ability to take a snapshot of a stuck volume
   • Send more frequent and detailed customer updates and staff developer support more widely during events
   • Develop tools that show through the APIs whether instances are impaired

   Lessons

   AWS writes that with more spare capacity the cluster would have absorbed the re-mirroring requests without a storm, and that it now understands how much capacity large recovery events need. The event also showed that the EBS control plane let a failure in one Availability Zone affect the others, so more work is needed to make zones behave as if completely independent. AWS adds that some customers had critical components in a single zone, and that multi-AZ deployments need to become easier to design and run.

   View case

By root cause2