IL-0350 IONOS Cloud · 2026-03-03networkingkubernetes
Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
The affected cluster in TXL runs two InfiniBand topologies. For a scheduled maintenance to replace a faulty switch, Topology 2 was disabled. The Multicast Forwarding Table (MFT) on all switches in Topology 1 was already strained by IPoIB multicast groups, so Topology 1 was running in a vulnerable state. A link flap on one switch flooded the InfiniBand control plane with alerts and triggered a fabric resweep, the automated mechanism that resets the network map. With the MFT at capacity the reprogramming failed, the surge of failed route updates overwhelmed the management layer and caused a critical gateway overload, BGP sessions dropped and hosts in the cluster lost connectivity. IONOS ended the network incident by switching from the affected control plane to its standby, rebooting the locked-up gateways and recovering Topology 1 from the scheduled maintenance.
While hosts were unreachable, every provisioning job for them was queued, and self-healing in the affected services kept adding jobs. After the network recovered, this volume caused resource locking that slowed the provisioning queue, so volume operations and self-healing waited and several services stayed degraded: IAM, and with it the DCD frontend, Managed Kubernetes and DBaaS. IONOS paused provisioning in the cluster, re-enabled it once connectivity improved, and let the backlog drain; provisioning and DBaaS were back to full operation at 21:49 UTC. IONOS named the coupling of services through the provisioning queue as a key reason the degradation lasted so long. The report says a related incident, covered by an RCA released on 11.03.2026, had the same technical root cause.
For the network, IONOS made the control-plane failover timers less aggressive and reduced heavy sweep activity, reduced multicast group memberships, and added monitoring with early alerts on forwarding table use and control-plane memory. Longer term it is moving the control plane to dedicated servers, auditing IPv6 and IPoIB driver settings and modernising the interconnect fabric, and it is changing IAM, Managed Kubernetes provisioning and the provisioning engine so that one cluster's backlog spills over less.
- Severity
- –
- Scope
- Network connectivity for hosts in one TXL cluster, with spillover to the DCD web frontend through IAM, the AI Model Hub, DBaaS, Managed Kubernetes and provisioning of infrastructure changes
- Users affected
- –
- Region
- TXL data center (Location DE/TXL)
- Time to detect
- –
- Time to resolve
- –
Symptoms
On 3 March 2026, during a scheduled maintenance to replace a faulty switch in a cluster in IONOS Cloud's TXL data center, monitoring alerts fired and the hosts in that cluster lost network connectivity. Customers saw intermittent network degradation or outages, and the Data Center Designer (DCD) web frontend and the AI Model Hub became unreachable.
In the network, a link flap on a switch in Topology 1, the topology still running while Topology 2 was disabled for the maintenance, flooded the InfiniBand control plane with alerts and started a fabric resweep. The Multicast Forwarding Table (MFT) on the Topology 1 switches was already near capacity from IPoIB multicast groups, so the reprogramming for the resweep failed. The flood of failed route updates overwhelmed the management layer, gateways locked up, BGP sessions dropped and the hosts in the cluster lost connectivity.
Each service showed the outage differently. Servers hosting DBaaS and Managed Kubernetes workloads lost BGP connectivity directly, and DBaaS was added to the affected services because of a rising error count. The AI Model Hub kept working but could not be reached, because its Managed Kubernetes cluster lost network connectivity. The DCD frontend was unavailable because the IAM service behind it could not be reached: an IP assignment maintenance job started during the incident stayed pending for a long time.
The disruption lasted much longer than the network fault. Provisioning jobs for resources in the cluster could not succeed and were queued, and services that entered self-healing added more jobs, so the queue grew exponentially. After the network recovered, the backlog caused resource locking that slowed the queue, and volume attach and detach operations and self-healing actions waited, which kept IAM and Managed Kubernetes degraded. At 12:38 UTC the status page reported problems again on the DCD frontend and AI Model Hub after an earlier recovery; provisioning in the cluster was paused at 12:54 UTC, AI Model Hub was operational again at 14:37 UTC, the DCD at 15:25 UTC, and provisioning and DBaaS were restored at 21:49 UTC.
Root cause
Root cause Capacity exhaustion
- Trigger
- During a scheduled maintenance to replace a faulty switch, Topology 2 of 2 in the affected cluster was disabled as planned, and a link flap on a switch in Topology 1 then flooded the InfiniBand control plane with alerts and started a fabric resweep.
Detection
IONOS began investigating monitoring alerts in TXL; the first status update was posted at 11:36 UTC.
Remediation
- Switched from the affected control plane to its standby
- Rebooted the locked-up gateways
- Recovered Topology 1 from the scheduled maintenance
- Paused provisioning in the cluster that had a backlog of queued jobs
- Re-enabled provisioning once network connectivity improved and watched job execution
- Re-established connectivity for the systems that support the DCD
Action items
- Adjusted the control-plane configuration to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table
- Reduced the number of multicast group memberships to relieve the forwarding table
- Set up continuous monitoring of forwarding table use and control-plane memory with alerts before capacity limits are reached
- Move the control plane to dedicated high-performance servers, separate from data-plane traffic
- Audit IPv6 configurations and IPoIB driver settings
- Modernise the interconnect fabric, including switches, gateways and drivers
- Hardened IAM maintenance protocols so that critical identity services keep working when the primary Cloud APIs are temporarily unavailable
- Move IAM to a dedicated platform without a dependency on the standard managed Kubernetes clusters
- Introduce a dedicated provisioning provider for DBaaS and Managed Kubernetes
- Add automated circuit breakers to the provisioning engine against overload and queue buildup from self-healing services
- Improve visibility and control over job execution and prioritisation in the provisioning queue
- Change the provisioning queue architecture to decouple services and reduce spillover
Lessons
IONOS identified the linking of services through the shared provisioning queue as a key reason the degradation lasted as long as it did. Alongside making each service more robust, it is decoupling the provisioning queue and addressing dependencies between services to reduce the blast radius of future incidents.
Sources
- https://status.ionos.cloud/incidents/3pyhtpglqf43 · official-status · archived copy
Related cases
IL-0450Oracle RAC commits behind order APIs kept waiting over 0.1 s on log file sync while average host CPU had headroomToss Securities · 2026same root cause · shared Database
IL-0553CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutesCircleCI · 2026same root cause · shared Database
IL-0379InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026same root cause · shared Database
IL-0309Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hoursPreset · 2026same root cause · shared Database
IL-0435Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flatCyberAgent · 2026same root cause · shared Database
IL-0497Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passedButtondown · 2026same root cause · shared Database
IL-0288ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026same root cause · shared Database
IL-0305Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026same root cause · shared Database
IL-0390PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalledPostHog · 2025same root cause · shared Database
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Database
IL-0287One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operationsMongoDB · 2025same root cause · shared Database
IL-0564Linode's Miami, Milan and Sao Paulo data centers had packet loss as traffic moved onto congested alternate pathsLinode · 2025same root cause · shared BGP routing
IL-0341Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hoursCloudflare · 2025same root cause · shared BGP routing
IL-0557Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hoursStorj · 2025same root cause · shared Database
IL-0551Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a dayAtlassian · 2025same root cause · shared Database
IL-0298Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disksWikimedia Foundation · 2025same root cause · shared Database
IL-0312Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went downGadget · 2024same root cause · shared Database
IL-0249Large training jobs on a cluster of over 10,000 GPUs slowed for tens of seconds to nearly ten hours without crashingAlibaba Group · 2024same root cause · shared InfiniBand fabric
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared Database
IL-0447An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baselineevery · 2024same root cause · shared Database
IL-0375MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a databaseMongoDB · 2024same root cause · shared Database
IL-0110A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Database
IL-0550Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pendingAtlassian · 2024same root cause · shared Database
IL-0270Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutesDiscord · 2023same root cause · shared Database
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Database
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared Database
IL-0326A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hoursMailPace · 2023same root cause · shared Database
IL-0546Buildkite jobs could not go to newly connected agents once agent IDs passed a 32-bit integer foreign key's limitBuildkite · 2023same root cause · shared Database
IL-0500UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up2i2c · 2022same root cause · shared Database
IL-0301Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice37signals · 2022same root cause · shared Database
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Database
IL-0302Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022GitHub · 2022same root cause · shared Database
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared Database
IL-0456Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQLInflab · 2022same root cause · shared Database
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021same root cause · shared Database
IL-0496Ably's Cassandra hit 100% CPU in every region and queries timed out globally, and the load returned on two more eveningsAbly · 2020same root cause · shared Database
IL-0443A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019same root cause · shared Database
IL-0037Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Database
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared Database
IL-0211A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hoursMailchimp · 2019same root cause · shared Database
IL-0064CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Database
IL-0300Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,64737signals · 2018same root cause · shared Database
IL-0334Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went downInstapaper · 2017same root cause · shared Database
IL-0333A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com downBuildkite · 2016same root cause · shared Database
IL-0282DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata serviceAmazon Web Services · 2015same root cause · shared Database
IL-0383PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming eventsPagerDuty · 2014 · sev1same root cause · shared Database
IL-0537Sakura Cloud host servers repeatedly lost their storage connection, about five minutes per port switchoverSakura Internet · 2012same root cause · shared InfiniBand fabric
IL-0320Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0350
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0350/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.