Cases11 cases

1 to 5 of 11

1. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table

   IONOS Cloud2026

   Symptoms

   On 3 March 2026, during a scheduled maintenance to replace a faulty switch in a cluster in IONOS Cloud's TXL data center, monitoring alerts fired and the hosts in that cluster lost network connectivity. Customers saw intermittent network degradation or outages, and the Data Center Designer (DCD) web frontend and the AI Model Hub became unreachable.

   Network unreachableLink flappingService outage

   Trigger

   During a scheduled maintenance to replace a faulty switch, Topology 2 of 2 in the affected cluster was disabled as planned, and a link flap on a switch in Topology 1 then flooded the InfiniBand control plane with alerts and started a fabric resweep.

   Root cause

   Capacity exhaustion

   Component

   InfiniBand fabricNetwork control plane (SDN)BGP routingNIC, cable and switch port linkModel hubDatabase

   Contributing factor

   Automation amplificationExcessive blast radius

   Detection

   IONOS began investigating monitoring alerts in TXL; the first status update was posted at 11:36 UTC.

   Remediation

   • Switched from the affected control plane to its standby
   • Rebooted the locked-up gateways
   • Recovered Topology 1 from the scheduled maintenance
   • Paused provisioning in the cluster that had a backlog of queued jobs
   • Re-enabled provisioning once network connectivity improved and watched job execution
   • Re-established connectivity for the systems that support the DCD

   Action items

   • Adjusted the control-plane configuration to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table
   • Reduced the number of multicast group memberships to relieve the forwarding table
   • Set up continuous monitoring of forwarding table use and control-plane memory with alerts before capacity limits are reached
   • Move the control plane to dedicated high-performance servers, separate from data-plane traffic
   • Audit IPv6 configurations and IPoIB driver settings
   • Modernise the interconnect fabric, including switches, gateways and drivers
   • Hardened IAM maintenance protocols so that critical identity services keep working when the primary Cloud APIs are temporarily unavailable
   • Move IAM to a dedicated platform without a dependency on the standard managed Kubernetes clusters
   • Introduce a dedicated provisioning provider for DBaaS and Managed Kubernetes
   • Add automated circuit breakers to the provisioning engine against overload and queue buildup from self-healing services
   • Improve visibility and control over job execution and prioritisation in the provisioning queue
   • Change the provisioning queue architecture to decouple services and reduce spillover

   Lessons

   IONOS identified the linking of services through the shared provisioning queue as a key reason the degradation lasted as long as it did. Alongside making each service more robust, it is decoupling the provisioning queue and addressing dependencies between services to reduce the blast radius of future incidents.

   View case
2. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down

   Magic Pages2025

   Symptoms

   On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

   Service outage

   Trigger

   One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

   Root cause

   Dependency failure

   Component

   DatabasePersistent volumes and stateful sets

   Contributing factor

   Harmful manual actionSingle point of failureMonitoring blind spot

   Remediation

   • Restarted the storage server to fix PVC provisioning
   • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
   • Cleaned up the corrupted cluster metadata
   • Redeployed MySQL Router

   Action items

   • Monitor the health of the storage system more closely
   • Improve the failover procedures for MySQL Router
   • Investigate and fix the status page so it shows reliable information

   Lessons

   The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

   View case
3. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover

   Honeycomb2023sev1

   Symptoms

   Standard alerting paged Honeycomb's on-call engineers about the performance and stability of event ingestion, and the automated load-shedding mechanisms tripped and sent further notifications. The status page described a higher than normal error rate and response time on the ingestion service.

   High error rateHigh latencyContainer crash loopService outageDatabase overload

   Trigger

   An abnormally large number of queries arrived from many separate connections within a few minutes.

   Root cause

   Capacity exhaustion

   Component

   DatabaseIn-memory cache

   Contributing factor

   Retry stormAutomation amplification

   Detection

   Standard alerting notified the on-call engineers about ingestion performance and stability, and the automated load-shedding mechanisms added notifications when they tripped. The first status post, about alerts on a higher than normal error rate and response time, went up at 18:28 PST (02:28 UTC), 13 minutes after the 02:15 UTC start.

   Remediation

   • Scaled the ingest fleet up manually and aggressively
   • Cut off most ingest for a few minutes so the ingest hosts could come back with enough capacity
   • Temporarily lowered the rate limits for the most aggressive traffic sources until their traffic subsided

   Action items

   • Keep the ingest fleet scaled up to prevent a repeat while longer-term fixes are worked on
   • Decide on the projects that will address these failure paths

   Lessons

   Load shedding and auto-scaling were in place, yet a burst that arrived faster than the cache could spread its writes still took the ingest hosts down. During recovery, Kubernetes' crash loop back-off worked against the operators: the hosts that had failed stayed offline while the fleet needed every host. Clients that retry whole OTel batches after a partial drop can multiply traffic many times over, so per-source rate limits were what finally stabilised ingest.

   View case
4. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast

   Microsoft Azure2023

   Symptoms

   From about 10:30 UTC on 30 August 2023, customers had trouble accessing or using Azure, Microsoft 365 and Power Platform services in Australia East. The affected Azure services included Azure Kubernetes Service (AKS) along with storage, SQL Database, Cosmos DB, virtual machines and Azure Resource Manager.

   Control plane unresponsiveHigh error rateService outage

   Trigger

   At 08:41 UTC a lightning strike on electrical infrastructure about 18 miles away caused a utility voltage sag that shut down the cooling chillers of datacenters in one Australia East availability zone; 13 chillers failed to restart automatically and five could not be restarted, so two data halls overheated and their infrastructure was shut down.

   Root cause

   Cloud provider outage

   Component

   AKS control planeDatabase

   Contributing factor

   Single point of failureExcessive blast radius

   Detection

   Cosmos DB impact was detected by monitoring at 10:47 UTC, first automated communications went to Azure Service Health at 10:48 UTC, the public Azure Status page was updated at 11:30 UTC, and all subscriptions in Australia East were sent portal communications at 11:36 UTC.

   Remediation

   • shut down the remaining networking, compute and storage in the two affected data halls so the chilled water loop could cool down
   • restarted the five chillers manually at 12:12 UTC
   • restored power to all hardware by 15:10 UTC and brought infrastructure back in phases
   • restored AKS operations through the recovery of the SQL Database operation queue, with no other mitigation

   Action items

   • AKS: convert all operation queue SQL Database databases to be zone redundant (estimated September 2023)
   • AKS: replace all cross-region SQL Database queue usage with zone-redundant Service Bus queues (estimated September 2023)
   • Change the control timing logic of the Chiller Management System so chillers do not fail to restart after voltage fluctuations
   • Change the emergency procedure for manual restarts of simultaneous chiller failures from an adjacent sequence to a load-based sequence per data hall
   • Increase technician staffing at the datacenter as a temporary measure for manual chiller restarts
   • Storage: optimise the large-scale recovery process and update troubleshooting guides (estimated December 2023)
   • Azure Service Fabric: improve reliability of SQL Database tenant ring recovery (estimated December 2023)
   • SQL Database: review the trigger criteria of auto-failover groups (estimated October 2023)
   • SQL Database: upgrade internal tooling to enable mass migration of databases (estimated December 2023)
   • Cosmos DB: optimise Service Managed Failover for single-region-write accounts (estimated November 2023)
   • ARM: complete the migration of its storage layer to the zonally redundant architecture (estimated December 2023)
   • Incident management: harden readiness, process and playbook for power-down scenarios (estimated October 2023)

   Lessons

   A control plane spread across availability zones still fails with a zone when one of its dependencies, here the SQL Database operation queue, is not zone redundant. When a second region uses the same database, the failure crosses into that region as well. Workloads already running were not expected to be affected; what stopped was everything that needs the managed service's control plane, such as scaling and cluster changes.

   View case
5. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic

   GitLab2022

   Symptoms

   GitLab's monitoring and alerting fired at once. One of the alerts reported a Patroni failover, and users of GitLab.com received 5XX errors; the CDN counted about 13 million requests answered with 503 during the outage.

   Service outageHigh error rateResources deleted

   Trigger

   An engineer ran a routine task with GitLab's tools in a local development environment; during a local deploy or rollback operation the tool's context changed from local development to production, and the change was applied to production.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   No safeguard on destructive change

   Detection

   GitLab's monitoring and alerting fired immediately; one of the alerts reported a Patroni failover.

   Remediation

   • re-applied the Consul release to the production Kubernetes cluster, with manual steps because the tooling did not support the redeploy directly

   Action items

   • Improve the Kubernetes deployment process to reduce the risk of running harmful deployment operations against production from local environments (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16875)
   • Integrate Teleport with the Kubernetes clusters, which could give SREs a read-only role by default (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16221)

   Lessons

   A tool that can switch its target from a local environment to production during a deploy or rollback turns routine local work into a production change. GitLab plans to improve its Kubernetes deployment process so that harmful deployment operations are less likely to be run against production from local environments, and to integrate Teleport with its clusters, which could give SREs a read-only role by default. Detection and diagnosis were fast because the alerts led straight from Patroni to its dependency, Consul. Bringing the deleted release back needed manual work, because the tooling did not make the redeploy straightforward, but it was still done relatively quickly.

   View case

By root cause5