Incident Library
Applied filters Root causeConfiguration changeSymptomService outageFactorIneffective rollbackClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
    Symptoms

    From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

    Service outageMonitoring outage
    Trigger

    From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

    Root cause
    Configuration change
    Component
    Message brokerDatabase
    Contributing factor
    Unrepresentative stagingNo staged rolloutIneffective rollback
    Detection

    Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

    Remediation
    • Rolled back the configuration change that over-restricted access to the metadata database
    • Identified and repaired every affected ordered subscription so the subscriptions received their backlog
    Action items
    • Enforce parity between pre-production and production environments so configuration changes are caught before production
    • Review change management so configuration changes roll out progressively according to their priority
    • Add monitoring that detects ordering metadata inconsistency
    • Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered
    View case
  2. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF
    OVHcloud2021
    Symptoms

    At 09:20 CET on 13 October 2021, during a planned change on a router, IPv4 traffic across OVHcloud's whole network stopped being routed, while all IPv6 traffic stayed reachable. The team detected a problem with the router's performance at 09:21 CET and escalated at once.

    Network unreachableService outage
    Trigger

    A planned change to a router in the Vint Hill datacentre in the United States, meant to strengthen anti-DDoS protection, included a command to regulate the redistribution of BGP into OSPF, and the router did not interpret that command correctly.

    Root cause
    Configuration change
    Component
    Interior routing (IGP)BGP routingBackbone network (WAN)
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    At 09:21 CET the team detected a problem with the router's performance and started the escalation process.

    Remediation
    • Disconnected the optical equipment to isolate the router's connectivity
    • Powered off the faulty router
    • Stabilised connectivity to restore the remaining services
    Action items
    • Re-evaluate the change procedure for this type of equipment, which applies the command line natively
    • Strengthen the related change process
    View case
  3. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
    Symptoms

    From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

    Service outageDatabase overloadHigh latencyHigh error rate
    Trigger

    A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

    Root cause
    Configuration change
    Component
    In-memory cacheDatabase
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    Google received an automated alert at 12:34 US/Pacific, one minute after the start.

    Remediation
    • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
    • Ramped Memcache traffic back up gradually while the error rate fell
    Action items
    • Reduce the number of readers of the global configuration to avoid contention during writes
    • Regionalise the configuration to limit the blast radius of similar failures
    View case
Check your cluster for this

These 3 cases involve Database and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change3 cases

Component
Database2Backbone network (WAN)1BGP routing1In-memory cache1Interior routing (IGP)1Message broker1
Contributing factor
Ineffective rollback3Excessive blast radius2No staged rollout1Unrepresentative staging1
Organisation
Google Cloud2OVHcloud1
  1. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
  2. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF
    OVHcloud2021
  3. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017