Incident Library
Applied filters SymptomScale-up failureFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped
    Google Cloud2024
    Symptoms

    From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

    Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate
    Trigger

    Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

    Root cause
    Software defect
    Component
    Network control plane (SDN)Cloud load balancerNAT gateway
    Contributing factor
    Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius
    Detection

    Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

    Remediation
    • Paused maintenance operations to prevent the shutdown from repeating
    • Brought the affected VPC Controllers back into a serving state in every zone
    • Stopped the automation responsible for the shutdown until safeguards are in place
    • Recovered partial Apigee proxy deployments by hand
    • Cleaned up VMs left behind by failed Dataproc cluster deletions
    Action items
    • Add safety measures that check that maintenance operations carry correctly scoped zone information
    • Make the underlying tools reject maintenance operations that do not specify zones
    • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
    • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures
    View case
  2. IL-0328 A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"
    CircleCI2021
    Symptoms

    From about 18:50 to 19:43 UTC on Monday, November 8, 2021, CircleCI customer jobs on all executors were blocked in a "not running" state. Errors and alerts appeared right after a deploy that changed a field type in the database of the job distribution service.

    Service outageHigh latencyScale-up failureCloud API rate limit exhausted
    Trigger

    At 18:50 UTC a deploy changed the type of a field in the PostgreSQL database of the job distribution service, in a way that was not backwards compatible.

    Root cause
    Faulty deployment
    Component
    DatabaseNode pool
    Contributing factor
    Ineffective rollbackExcessive blast radiusGap in automated tests
    Detection

    Errors and alerts related to the distribution failure appeared immediately after the 18:50 UTC deploy.

    Remediation
    • Rolled back the field type change, which did not clear the errors
    • Wrote, built and manually deployed a change that made the distribution system ignore the field
    • Manually scaled the primary compute cluster for the expected thundering herd of work
    • Monitored until processing was normal and resolved the incident
    Action items
    • Fix the gaps found in the testing strategy
    • Change how the distributor reacts to unexpected data so that this kind of deploy should not affect all jobs
    • Make recovery faster when unforeseen failures occur
    View case
Check your cluster for this

These 2 cases involve Cloud load balancer and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Software defect1 case

Component
Cloud load balancerNAT gatewayNetwork control plane (SDN)
Contributing factor
Excessive blast radiusChange with a delayed effectIgnored early signalNo safeguard on destructive change
Organisation
Google Cloud
  1. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped
    Google Cloud2024

Faulty deployment1 case

Component
DatabaseNode pool
Contributing factor
Excessive blast radiusIneffective rollbackGap in automated tests
Organisation
CircleCI
  1. IL-0328 A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"
    CircleCI2021