Incident Library
Applied filters Root causeSoftware defectDomainKubernetesOrganisationPinterestClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0247 One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to Kubernetes
    Pinterest2025
    Symptoms

    During performance testing of Pinterest's search system Manas on its new Kubernetes platform, a small but persistent share of search requests timed out. About one request in a million took 100 times longer than usual.

    High latencyIntermittent timeouts
    Root cause
    Software defect
    Component
    Telemetry agentLinux kernel
    Contributing factor
    Unreviewed defaults
    Detection

    Performance testing with traffic sent to a test cluster showed a small but persistent percentage of requests timing out.

    Remediation
    • Disabled cAdvisor's working set size estimation on all PinCompute nodes
    Lessons

    CPU shielding does not fully isolate a process from its neighbours. Showing early that the node image was not the cause narrowed the search and avoided dead ends. Simple blackbox debugging, breaking things until the system stops working, works as a way of bisecting the problem.

    View case
Check your cluster for this

This case involves Linux kernel and Telemetry agent. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Linux kernelTelemetry agent
Contributing factor
Unreviewed defaults
Organisation
Pinterest
  1. IL-0247 One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to Kubernetes
    Pinterest2025