Incident Library
Applied filters Root causeOperator errorSymptomNetwork unreachableFactorNo safeguard on destructive changeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0349 IONOS Private Cloud in Berlin lost management access, storage and VM internet routing after a VLAN change
    IONOS Cloud2026
    Symptoms

    On 15 January 2026, network services for VMware-based Private Clouds in IONOS Cloud's Berlin location stopped working: customers lost management VPN access and traffic to and from their virtual machines. Inside IONOS, several management interfaces in the Berlin region could no longer be reached.

    Network unreachableStorage disconnected or stale mounts
    Trigger

    During a planned maintenance window to add a new VLAN to the management devices, a human error in the command replaced the existing VLANs on the trunk between the core and management devices instead of adding to them, so only the new VLAN stayed active.

    Root cause
    Operator error
    Component
    Network switchTransit gateway or network interconnect
    Contributing factor
    Excessive blast radiusNo safeguard on destructive change
    Remediation
    • Restarted the affected storage services in the final stage of recovery, which caused a brief second outage
    Action items
    • Audit the management device architecture to segment it better (target Q3 2026)
    • Evaluate and implement ways to prevent unintended VLAN changes, such as improved configuration safeguards or automation scripts (target Q1 2026)
    View case
  2. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
    Symptoms

    On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

    Service outageNetwork unreachableResources deleted
    Trigger

    At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

    Root cause
    Operator error
    Component
    TerraformCI systemVPC route tables and firewall rules
    Contributing factor
    No safeguard on destructive change
    Detection

    High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

    Remediation
    • Ran terraform apply locally to rebuild the subnet route table associations
    Action items
    • Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
    • Find a way to hold queued pull requests while manual infrastructure updates are under way
    • Find ways to isolate the plan and apply tasks of infrastructure pipelines better
    View case
Check your cluster for this

These 2 cases involve CI system and Transit gateway or network interconnect. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error2 cases

Component
CI system1Transit gateway or network interconnect1Network switch1Terraform1VPC route tables and firewall rules1
Contributing factor
No safeguard on destructive change2Excessive blast radius1
Organisation
IONOS Cloud1UK Ministry of Justice1
  1. IL-0349 IONOS Private Cloud in Berlin lost management access, storage and VM internet routing after a VLAN change
    IONOS Cloud2026
  2. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024