Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases7 cases

6 to 7 of 7

  1. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF
    OVHcloud2021
    Symptoms

    At 09:20 CET on 13 October 2021, during a planned change on a router, IPv4 traffic across OVHcloud's whole network stopped being routed, while all IPv6 traffic stayed reachable. The team detected a problem with the router's performance at 09:21 CET and escalated at once.

    Network unreachableService outage
    Trigger

    A planned change to a router in the Vint Hill datacentre in the United States, meant to strengthen anti-DDoS protection, included a command to regulate the redistribution of BGP into OSPF, and the router did not interpret that command correctly.

    Root cause
    Configuration change
    Component
    Interior routing (IGP)BGP routingBackbone network (WAN)
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    At 09:21 CET the team detected a problem with the router's performance and started the escalation process.

    Remediation
    • Disconnected the optical equipment to isolate the router's connectivity
    • Powered off the faulty router
    • Stabilised connectivity to restore the remaining services
    Action items
    • Re-evaluate the change procedure for this type of equipment, which applies the command line natively
    • Strengthen the related change process
    View case
  2. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
    Symptoms

    From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

    Service outageDatabase overloadHigh latencyHigh error rate
    Trigger

    A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

    Root cause
    Configuration change
    Component
    In-memory cacheDatabase
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    Google received an automated alert at 12:34 US/Pacific, one minute after the start.

    Remediation
    • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
    • Ramped Memcache traffic back up gradually while the error rate fell
    Action items
    • Reduce the number of readers of the global configuration to avoid contention during writes
    • Regionalise the configuration to limit the blast radius of similar failures
    View case
Check your cluster for this

These 7 cases involve BGP routing and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change7 cases

Component
BGP routing3Database3Message broker2Backbone network (WAN)1In-memory cache1Automated capacity manager1Interior routing (IGP)1Linux kernel1Telemetry agent1
Contributing factor
Ineffective rollback7Excessive blast radius2Automation amplification1Circular dependency1Unreviewed defaults1Limited debug access1Monitoring blind spot1No configuration tests1No staged rollout1Overlapping failures1Retry storm1Missed in review1Unrepresentative staging1
Organisation
Google Cloud2Autumn1Clever Cloud1OVHcloud1PSBA1Wikimedia Foundation1
  1. IL-0400 Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default route
    PSBA2026
  2. IL-0487 Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexed
    Autumn2026
  3. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
  4. IL-0285 Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panics
    Clever Cloud2024
  5. IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs
    Wikimedia Foundation2021
  6. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF
    OVHcloud2021
  7. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017