Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Partial rollout

부분 배포 · partial-rollout

3 cases · 2019 to 2024

Root cause Configuration change1 Dependency failure1 Software defect1

Component Cloud load balancer1 GPU driver1 Model inference server1 InfiniBand fabric1

Symptom High error rate2 GPU unavailable1 High latency1 Intermittent timeouts1

Organisation Google Cloud1 Grafana Labs1 Imbue1

Cases

IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024
Root cause Dependency failure
IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the loadGoogle Cloud · 2024
Root cause Software defect
IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down

View on the graph

Pod priority and preemption1

Service outage1

Throughput degradation1

Training job failure1

Grafana Labs · 2019

Root cause Configuration change