Cases6 cases

1 to 5 of 6

1. IL-0349 IONOS Private Cloud in Berlin lost management access, storage and VM internet routing after a VLAN change

   IONOS Cloud2026

   Symptoms

   On 15 January 2026, network services for VMware-based Private Clouds in IONOS Cloud's Berlin location stopped working: customers lost management VPN access and traffic to and from their virtual machines. Inside IONOS, several management interfaces in the Berlin region could no longer be reached.

   Network unreachableStorage disconnected or stale mounts

   Trigger

   During a planned maintenance window to add a new VLAN to the management devices, a human error in the command replaced the existing VLANs on the trunk between the core and management devices instead of adding to them, so only the new VLAN stayed active.

   Root cause

   Operator error

   Component

   Network switchTransit gateway or network interconnect

   Contributing factor

   Excessive blast radiusNo safeguard on destructive change

   Remediation

   • Restarted the affected storage services in the final stage of recovery, which caused a brief second outage

   Action items

   • Audit the management device architecture to segment it better (target Q3 2026)
   • Evaluate and implement ways to prevent unintended VLAN changes, such as improved configuration safeguards or automation scripts (target Q1 2026)

   View case
2. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline

   UK Ministry of Justice2024

   Symptoms

   On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

   Service outageNetwork unreachableResources deleted

   Trigger

   At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

   Root cause

   Operator error

   Component

   TerraformCI systemVPC route tables and firewall rules

   Contributing factor

   No safeguard on destructive change

   Detection

   High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

   Remediation

   • Ran terraform apply locally to rebuild the subnet route table associations

   Action items

   • Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
   • Find a way to hold queued pull requests while manual infrastructure updates are under way
   • Find ways to isolate the plan and apply tasks of infrastructure pipelines better

   View case
3. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case
4. IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers

   Acquia2019

   Symptoms

   On 7 February 2019 from 22:17 to 23:25 UTC, some Acquia Cloud Enterprise sites in US-East became unstable or went down for multiple customers. The web servers could not reach the Memcache servers, PHP processes on the web servers were blocked, and the sites became unavailable.

   Service outageNetwork unreachable

   Trigger

   While setting up a temporary file system to store diagnostic data, a rule was deleted by mistake from a VPC security group.

   Root cause

   Operator error

   Component

   VPC route tables and firewall rulesIn-memory cache

   Contributing factor

   Ignored early signal

   Detection

   Acquia's configuration drift alerting fired, but the alert was not seen in time to shorten the outage.

   Remediation

   • Restored the missing rule in the security group by hand

   Action items

   • Correct the internal workflow with all involved teams to prevent similar mistakes
   • Alert internal teams faster about configuration drift in VPC security groups

   View case
5. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of data

   Hatena2017

   Symptoms

   On 7 August 2017, while checking the application after the maintenance that moved Mackerel from an on-premises data center to AWS, the team found that data updated during a specific time window was missing. Everything written to the PostgreSQL database between 14:30 and 15:30 JST was gone.

   Data lossReplication lagNetwork unreachable

   Trigger

   One hour after the application was switched to the newly promoted primary db001, db001 lost network connectivity and the database failed over to the standby db002.

   Root cause

   Operator error

   Component

   DatabaseDatabase replicationDatabase failover manager

   Contributing factor

   Overlapping failuresUnreviewed defaultsUnrepresentative stagingAutomation amplification

   Detection

   The team found the missing data while checking how the application behaved after the migration work.

   Remediation

   • Tried to restore the lost updates by hand from the table data left on db001 and gave up
   • Tried point-in-time recovery from the base backup and WAL and gave up
   • Built a new replica db003 with db002 as primary to restore redundancy
   • Kept db001 in maintenance state for investigation
   • Watched db002 for some time to confirm the problem did not recur, then completed the migration

   Action items

   • Change the PostgreSQL replica creation script to set recovery_target_timeline = 'latest' in recovery.conf by default
   • Add a check to a server configuration test tool such as Serverspec that recovery.conf contains recovery_target_timeline = 'latest'

   Lessons

   A setting needed only for a rare layout, such as a grandchild replica during a migration, was applied by hand through a script option and was forgotten. Because recovery_target_timeline = 'latest' is harmless on ordinary standbys too, Hatena now sets it by default whenever a replica is created. The staging rehearsal used the same layout and steps but had the setting right, so it could not catch the mistake. Hatena notes that a similar layout may be used again for a future PostgreSQL upgrade or a move to another region.

   View case

By root cause1