Incident Library
Applied filters Root causeSoftware defectFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
    Symptoms

    From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.

    High error rateResources deletedService outage
    Trigger

    A faulty resource cleanup process marked Vertex Prediction Endpoints as deleted, globally.

    Root cause
    Software defect
    Component
    Model request routing
    Contributing factor
    Automation amplificationExcessive blast radius
    Detection

    The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.

    Remediation
    • restored the Vertex Prediction Endpoints that had been marked as deleted
    View case
Check your cluster for this

These 6 cases involve Node operating system image and Automated capacity manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect6 cases

Component
Node operating system image3Automated capacity manager1Cloud load balancer1Application database1Authoritative DNS servers and records1GPU driver1Karpenter1kubelet1Node launch template1Linux kernel1Model request routing1Node pool1Terraform1AWS VPC CNI1
Contributing factor
Automation amplification6Monitoring blind spot2Unverified assumption1Excessive blast radius1Change with a delayed effect1Cache-masked failure1No health probes1No runbook1No safeguard on destructive change1No staged rollout1Overlapping failures1Retry storm1
Organisation
Amazon Web Services1DFDS1Google Cloud1Inflab1Microsoft1OpenAI1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  3. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False
    Inflab2025
  4. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries
    Microsoft2025
  5. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  6. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022