Incident Library
Applied filters ComponentCNI pluginFactorOverlapping failuresOrganisationincident.ioClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes
    incident.io2023
    Symptoms

    Shortly after incident.io moved its infrastructure fully into Google Cloud, the application saw a larger-than-normal number of Postgres connection timeouts, and new database connections per second were in the hundreds, spiking up and down. At the same time, the memcached instance used for transient data and faster API responses had sporadic connection timeouts.

    Intermittent timeoutsPacket loss
    Trigger

    After the move to Google Cloud, the application opened hundreds of new Postgres connections per second, and an accidental database join made it send thousands of duplicate calls to an external third party in parallel.

    Root cause
    Capacity exhaustion
    Component
    CNI pluginDatabaseIn-memory cache
    Contributing factor
    Unreviewed defaultsOverlapping failures
    Detection

    An incident was declared when the number of Postgres connection timeouts rose above normal. The remaining cache timeouts were traced through the logs of the Kubernetes node that served an affected request, the request's trace and the node's metrics.

    Remediation
    • Doubled the maximum Postgres connection lifespan from 15m to 30m
    • Made the Postgres connection pools static so they always keep N connections open
    • Raised the memcache client's MaxIdleConns from 2 to 20
    • Moved temporarily from Google's hosted memcached to a memcached inside the Kubernetes cluster
    • Shipped the fix for the accidental database join that made thousands of duplicate external calls
    Action items
    • Set more sensible concurrency limits for outbound network calls
    Lessons

    Network timeouts are not always local to one request or pod: when they appear, the whole activity of the node, including other pods on it, needs to be checked. Staying within a provider's rate limits does not keep outbound calls from affecting a node, so outbound network calls need more sensible concurrency limits.

    View case
Check your cluster for this

This case involves In-memory cache and CNI plugin. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
In-memory cacheCNI pluginDatabase
Contributing factor
Unreviewed defaultsOverlapping failures
Organisation
incident.io
  1. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes
    incident.io2023