Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0293 Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet loss
    Google Cloud2023
    Symptoms

    From 19:50 US/Pacific on 4 December 2023, Cloud Bigtable customers in us-central1 saw higher latency and a low level of connection errors, affecting about 0.06% of Bigtable dataplane requests. Automated monitoring first detected the problem at 23:35.

    High latencyHigh error ratePacket lossIntermittent connection failures
    Trigger

    A change that caused an isolation failure on the machines running Cloud Bigtable servers.

    Root cause
    Faulty deployment
    Component
    Database
    Contributing factor
    –
    Detection

    Automated monitoring detected the problem at 23:35 US/Pacific on 4 December 2023.

    Remediation
    • Redirected some traffic away from a subset of the affected machines
    • Rolled back the change that triggered the isolation failure
    Action items
    • Put controls in place to prevent a recurrence
    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Database
Contributing factor
–
Organisation
Google Cloud
  1. IL-0293 Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet loss
    Google Cloud2023