Incident Library
Applied filters SymptomPacket lossFactorUnreviewed defaultsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets
    Cybozu2024
    Symptoms

    A team using Neco, the Kubernetes platform run by Cybozu's Cloud Platform team, reported what looked like a performance problem in the cluster DNS servers. An application in a pod that resolved cluster Services at hundreds of requests per second frequently got `no such host` and `i/o timeout` errors.

    DNS resolution failuresIntermittent timeoutsPacket loss
    Trigger

    The number of MySQL clusters watched by the monitoring application increased, and after that the name resolution errors became frequent.

    Root cause
    Capacity exhaustion
    Component
    Linux kernelClient-side DNS resolver and cache
    Contributing factor
    Unreviewed defaults
    Detection

    A team using the platform contacted the Cloud Platform team, describing the errors as a performance problem in the cluster DNS server.

    Remediation
    • Raised the kernel parameter netdev_max_backlog on the nodes
    • Set up alerts on softnet
    • Wrote a runbook for the softnet alerts
    Action items
    • Evaluate Cilium's netkit device so pod packets bypass the backlog queue
    Lessons

    Raising netdev_max_backlog makes the same failure less likely and the new alerts catch it early, but it is not a root fix: a pod that sends more packets can fill the backlog queue again. Cybozu sees replacing the veth interface with netkit, which skips the backlog queue, as the root fix.

    View case
  2. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes
    incident.io2023
    Symptoms

    Shortly after incident.io moved its infrastructure fully into Google Cloud, the application saw a larger-than-normal number of Postgres connection timeouts, and new database connections per second were in the hundreds, spiking up and down. At the same time, the memcached instance used for transient data and faster API responses had sporadic connection timeouts.

    Intermittent timeoutsPacket loss
    Trigger

    After the move to Google Cloud, the application opened hundreds of new Postgres connections per second, and an accidental database join made it send thousands of duplicate calls to an external third party in parallel.

    Root cause
    Capacity exhaustion
    Component
    CNI pluginDatabaseIn-memory cache
    Contributing factor
    Unreviewed defaultsOverlapping failures
    Detection

    An incident was declared when the number of Postgres connection timeouts rose above normal. The remaining cache timeouts were traced through the logs of the Kubernetes node that served an affected request, the request's trace and the node's metrics.

    Remediation
    • Doubled the maximum Postgres connection lifespan from 15m to 30m
    • Made the Postgres connection pools static so they always keep N connections open
    • Raised the memcache client's MaxIdleConns from 2 to 20
    • Moved temporarily from Google's hosted memcached to a memcached inside the Kubernetes cluster
    • Shipped the fix for the accidental database join that made thousands of duplicate external calls
    Action items
    • Set more sensible concurrency limits for outbound network calls
    Lessons

    Network timeouts are not always local to one request or pod: when they appear, the whole activity of the node, including other pods on it, needs to be checked. Staying within a provider's rate limits does not keep outbound calls from affecting a node, so outbound network calls need more sensible concurrency limits.

    View case
  3. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets
    SEEK2021
    Symptoms

    In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.

    High error ratePacket lossIntermittent timeouts
    Trigger

    In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.

    Root cause
    Configuration change
    Component
    Transit gateway or network interconnect
    Contributing factor
    Unknown limitUnreviewed defaultsMonitoring blind spot
    Detection

    The team noticed soon after the routing change that search on seek.com.au was not working.

    Remediation
    • Rolled back the migration of the search traffic's peering connection to the transit gateway
    Action items
    • Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up
    Lessons

    A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.

    View case
Check your cluster for this

These 3 cases involve In-memory cache and CNI plugin. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Capacity exhaustion2 cases

Component
In-memory cache1CNI plugin1Database1Client-side DNS resolver and cache1Linux kernel1
Contributing factor
Unreviewed defaults2Overlapping failures1
Organisation
Cybozu1incident.io1
  1. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets
    Cybozu2024
  2. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes
    incident.io2023

Configuration change1 case

Component
Transit gateway or network interconnect
Contributing factor
Unreviewed defaultsUnknown limitMonitoring blind spot
Organisation
SEEK
  1. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets
    SEEK2021