Graph
Cases3 cases
1 to 3 of 3
IL-0350Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud2026- Symptoms
On 3 March 2026, during a scheduled maintenance to replace a faulty switch in a cluster in IONOS Cloud's TXL data center, monitoring alerts fired and the hosts in that cluster lost network connectivity. Customers saw intermittent network degradation or outages, and the Data Center Designer (DCD) web frontend and the AI Model Hub became unreachable.
- Trigger
During a scheduled maintenance to replace a faulty switch, Topology 2 of 2 in the affected cluster was disabled as planned, and a link flap on a switch in Topology 1 then flooded the InfiniBand control plane with alerts and started a fabric resweep.
- Root cause
- Component
- Contributing factor
- Detection
IONOS began investigating monitoring alerts in TXL; the first status update was posted at 11:36 UTC.
- Remediation
- Switched from the affected control plane to its standby
- Rebooted the locked-up gateways
- Recovered Topology 1 from the scheduled maintenance
- Paused provisioning in the cluster that had a backlog of queued jobs
- Re-enabled provisioning once network connectivity improved and watched job execution
- Re-established connectivity for the systems that support the DCD
- Action items
- Adjusted the control-plane configuration to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table
- Reduced the number of multicast group memberships to relieve the forwarding table
- Set up continuous monitoring of forwarding table use and control-plane memory with alerts before capacity limits are reached
- Move the control plane to dedicated high-performance servers, separate from data-plane traffic
- Audit IPv6 configurations and IPoIB driver settings
- Modernise the interconnect fabric, including switches, gateways and drivers
- Hardened IAM maintenance protocols so that critical identity services keep working when the primary Cloud APIs are temporarily unavailable
- Move IAM to a dedicated platform without a dependency on the standard managed Kubernetes clusters
- Introduce a dedicated provisioning provider for DBaaS and Managed Kubernetes
- Add automated circuit breakers to the provisioning engine against overload and queue buildup from self-healing services
- Improve visibility and control over job execution and prioritisation in the provisioning queue
- Change the provisioning queue architecture to decouple services and reduce spillover
- Lessons
IONOS identified the linking of services through the shared provisioning queue as a key reason the degradation lasted as long as it did. Alongside making each service more robust, it is decoupling the provisioning queue and addressing dependencies between services to reduce the blast radius of future incidents.
- Symptoms
On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.
- Root cause
- Component
- Contributing factor
- Remediation
- Replaced the two malfunctioning L4 devices with higher-performance devices, tested them and resumed Government24 at 09:00 on 18 November
- Moved the connection on the defective port of the Daejeon center router to another port at 07:00 on 19 November
- Swapped out the module of the router that had the fault
- Action items
- Inspect all old equipment that may have similar port defects
- Revise the incident handling manual so the public is told about an outage quickly
- Announce the status of service recovery quickly through various means
- Build integrated monitoring of critical service systems and related equipment to catch signs of failure early
- Form a standing incident response team to send into major outages
- Prepare administrative measures and a response manual so administrative services can still be provided when core digital government services stop
- Name a paralysis of the national computer network as a type of disaster and accident, managed from prevention to recovery
- Build a government-wide crisis response system for linked information systems and strengthen the control tower for outage prevention and response
- Improve how public IT projects are run and managed, replace hardware and software past their service life quickly and make project pricing realistic
- Reduce the reliance on outsourcing for developing and operating information systems and bring digital talent into government
- Review the redundancy, recovery systems and network configuration of the systems in the National Information Resources Service, and its organisation
- Work with manufacturers on a way to find and monitor failures of internal parts earlier
- Lessons
The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.
IL-0263GitHub was slow and erroring all day and down for 18 minutes as links between access and aggregation switches saturatedGitHub2012- Symptoms
On Friday, 30 November 2012, GitHub had sporadic bursts of slow responses and intermittent errors for the whole day and was completely unavailable for 18 minutes. One fileserver pair was offline for a large part of the day, which affected a small percentage of repositories; this was a side effect of the network problems on the high-availability clustering between the fileserver nodes.
- Trigger
On the morning of 30 November GitHub moved the A switches in its server cabinets onto the new aggregation switches, the second half of a migration whose B-switch half had been done the day before.
- Root cause
- Component
- Contributing factor
- Detection
Engineers saw the problems during the maintenance itself: connectivity to the B switch was lost as the last A switch was connected, and afterwards performance stayed low and the links between access and aggregation switches showed as completely saturated.
- Remediation
- Fixed the bridge-loop problem and returned the affected B switch to service
- Removed the partial link failure detection setting from the access switches
- Escalated to the network vendor, which raised the case to its highest severity level
- Restarted core processes on the aggregation switches in a Saturday 0600 Pacific maintenance window so they learned MAC addresses again
- Action items
- Deploy the vendor's software update that fixes the MAC learning issue to the switches
- Closely monitor aggregation-to-access capacity and keep a workaround process ready if the problem returns
- Build a duplicate of the network stack, from the routers through the access switches, as a staging environment for testing such changes
- Add automated network monitoring that alerts sooner on similar issues
- Run scheduled incident response exercises with scenarios that counter tunnel vision
- Contact the network vendor's support at the first sign of trouble in the network
- Lessons
GitHub had designed the maintenance to have no impact on customers and clearly failed, so it plans to invest in a staging copy of its network to test such changes and catch bugs like this one. During the response the team fixated for a long time on a bridge loop, which kept it from considering other causes. GitHub had a very positive experience with the vendor's support staff, so it will contact that support team at the first sign of network trouble.
These 3 cases involve NIC, cable and switch port link and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause3
Capacity exhaustion1 case
- Component
- Contributing factor
- Organisation
Hardware failure1 case
- Component
- Contributing factor
- Organisation
Software defect1 case
- Contributing factor
- Organisation