Incident Library

IL-0386 incident.io · 2023-09-26 (published)kubernetesgcpdatabasenetworking

After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes

incident.io runs a simple GKE setup, and the only relevant configurable behaviour of the cluster was that Dataplane V2 was turned on. After the move into Google Cloud, Postgres connection timeouts rose, and the memcached cache also had sporadic timeouts. The memcache client used a 100ms network timeout, so any dropped TCP packet was likely to end in a timeout, and it kept the default of 2 idle connections.

The team first doubled the Postgres maximum connection lifespan and made the connection pools static, which brought new connections down to about 1/s, and then raised MaxIdleConns from 2 to 20, which reduced the cache timeouts. Meanwhile an unrelated accidental database join had been spawning thousands of goroutines that called an external provider; when its fix landed, the database and cache errors nearly stopped. The remaining cache timeouts happened when a single request set a few hundred cache values at once while also generating many Google Cloud Storage URLs, and at those moments the node showed a spike in new connections and dropped packets.

The GKE documentation states that the Dataplane V2 agent, anetd, can consume significant CPU when many TCP connections are opened and closed rapidly on a node, and recommends keep-alives and connection pooling. incident.io found that this matched everything seen so far: it judged it likely that its own bursts of short-lived connections kept anetd on the node very busy for a couple of seconds at a time, and each fix that reduced new connections had helped, while the problem persisted, less often, whenever many external requests ran in parallel. The team decided not to turn off Dataplane V2, because that would require a new cluster and would only hide a pattern the application itself was causing, and instead planned to limit outbound network concurrency.

Severity
–
Scope
A small share of requests hit Postgres connection timeouts and memcached i/o timeouts; at times health checks on an affected node failed
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

Shortly after incident.io moved its infrastructure fully into Google Cloud, the application saw a larger-than-normal number of Postgres connection timeouts, and new database connections per second were in the hundreds, spiking up and down. At the same time, the memcached instance used for transient data and faster API responses had sporadic connection timeouts.

The cache error read "saving value to cache: setting cache value: read tcp 10.16.4.33:35586->10.67.0.5:11211: i/o timeout". It appeared most often when caching the list of timeline events for an incident, and only for a tiny handful of requests among hundreds of thousands, with no consistent pattern in users, organizations, nodes or requests. On the Kubernetes node that served one failing request, a separate request also failed its health check because it could not reach the cache.

The errors changed as the team made fixes. Doubling the maximum Postgres connection lifespan from 15m to 30m and making the connection pools static cut new connections from about 200/s to about 1/s. Raising the memcache client's MaxIdleConns from 2 to 20 helped at once, but a few days later many more "read i/o" timeouts appeared. Moving from Google's hosted memcached to a memcached inside the Kubernetes cluster made little difference. When a fix for an unrelated bad database join, which had been making thousands of duplicate calls to an external third party, reached production, the database and cache errors nearly disappeared, but some cache errors remained.

The first theory was a race in which goroutines reused an expiring connection and then opened new connections en masse. The team also noticed that pooled memcache connections were never checked for health, but did not think that was what was happening. The trace of an affected request showed a few hundred cache set operations at the same moment, together with many calls to generate Google Cloud Storage URLs, with call durations rising from about 1 ms by a few orders of magnitude. Node metrics showed a spike in new connections and more dropped packets on that node, which pointed at the node as a whole rather than one process. The GKE documentation says the Dataplane V2 agent pods (anetd) can use two or three vCPUs when many TCP connections are opened and closed quickly on a node; the anetd logs showed nothing obvious, but its CPU use had jumped.

Symptom Intermittent timeoutsPacket loss

Root cause

Root cause Capacity exhaustion

Component CNI pluginDatabaseIn-memory cache
Contributing factor Unreviewed defaultsOverlapping failures
Trigger
After the move to Google Cloud, the application opened hundreds of new Postgres connections per second, and an accidental database join made it send thousands of duplicate calls to an external third party in parallel.

Detection

An incident was declared when the number of Postgres connection timeouts rose above normal. The remaining cache timeouts were traced through the logs of the Kubernetes node that served an affected request, the request's trace and the node's metrics.

Remediation

Action items

Lessons

Network timeouts are not always local to one request or pod: when they appear, the whole activity of the node, including other pods on it, needs to be checked. Staying within a provider's rate limits does not keep outbound calls from affecting a node, so outbound network calls need more sensible concurrency limits.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CNI plugin, Database, In-memory cache among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.