Incident Library

IL-0293 Google Cloud · 2023-12-04gcpdatabasenetworking

Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet loss

The root cause was an isolation failure on the machines that ran Cloud Bigtable servers in us-central1, set off by a change, and it showed mainly as a low rate of network packet loss. Automated monitoring detected the problem at 23:35 on 4 December, 3 hours 45 minutes after the 19:50 start, and engineers began investigating.

At 14:40 on 5 December engineers redirected some traffic away from a subset of the affected machines, which partly mitigated the impact; the report separately gives the partial mitigation as 15 hours after the start. Engineers then identified the change that triggered the isolation failure and rolled it back, and the latency was fully resolved at 03:55 on 6 December, 1 day, 8 hours and 5 minutes after the start. Google said it put controls in place to prevent a recurrence. The status page marked the issue resolved for all affected users at 08:23 on 6 December.

Severity
–
Scope
About 0.06% of Cloud Bigtable dataplane requests had higher latency and errors; Google Cloud Tasks was also listed as affected
Users affected
Bigtable customers in us-central1, for longest those running non-replicated clusters
Region
us-central1
Time to detect
3h 45m
Time to resolve
32h 5m

Symptoms

From 19:50 US/Pacific on 4 December 2023, Cloud Bigtable customers in us-central1 saw higher latency and a low level of connection errors, affecting about 0.06% of Bigtable dataplane requests. Automated monitoring first detected the problem at 23:35.

The first status updates on 5 December described a Google Cloud Networking issue in us-central1 that affected Bigtable and Google Kubernetes Engine. Later updates said the cause was not Google Cloud Networking and that there was no impact specific to Google Kubernetes Engine. Google Cloud Tasks was also listed as affected.

After a partial mitigation on 5 December, the problem was resolved for most customers, but customers running non-replicated Bigtable clusters in us-central1 kept seeing elevated errors and latency until early on 6 December. The status page offered no workaround.

Symptom High latencyHigh error ratePacket lossIntermittent connection failures

Root cause

Root cause Faulty deployment

Component Database
Contributing factor –
Trigger
A change that caused an isolation failure on the machines running Cloud Bigtable servers.

Detection

Automated monitoring detected the problem at 23:35 US/Pacific on 4 December 2023.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.