Incident Library

IL-0247 Pinterest · 2025-07-16 (published)kubernetesaws

One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to Kubernetes

Manas is Pinterest's in-house search system; it serves dozens of search indices from more than 100 clusters on thousands of hosts. In 2024 Pinterest built a new setup for Manas on PinCompute, its in-house Kubernetes platform, and in early 2025 began performance testing it. A root node fans each request out to leaf nodes, and each leaf serves one shard of an index with memory-mapped lookups during retrieval and ranking. On the test cluster a small but persistent share of requests timed out because leaf max latency spiked as high as 5s, and Pinterest paused the migration to find the cause.

Changing the hardware, node image, cgroup limits and container setup and isolating the CPUs did not help. Suspending processes one by one showed that cAdvisor caused the spikes, and perf showed it spending its time in the kernel's smaps feature. The cause was the metric container_referenced_bytes, which cAdvisor enables by default and whose documentation marks it as intrusive to collect. It estimates working set size by scanning the whole page table, counting accessed bits and then clearing every one of them, and PinCompute ran cAdvisor every 30 seconds. Manas leaf indices can be hundreds of gigabytes, and a host with 100GB of used memory can have 25 million page table entries; smaps and clear_refs each take locks while scanning all of the kernel's page structures, and clear_refs also flushes the working set from the TLB; Pinterest concluded that this memory contention was the likely cause of the stalls in the memory-intensive leaf process. Pinterest turned off cAdvisor's working set estimation on all PinCompute nodes with a one-line change and opened an issue in the cAdvisor repository. cAdvisor itself could not be removed because PinCompute needs its container metrics, including for workload autoscaling.

Severity
–
Scope
Manas test search cluster on PinCompute during the migration to Kubernetes; the migration was paused
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

During performance testing of Pinterest's search system Manas on its new Kubernetes platform, a small but persistent share of search requests timed out. About one request in a million took 100 times longer than usual.

Across the leaf nodes, p9999 and lower latencies stayed in the normal range, but max latency reached up to 5s where under 60ms was expected. On a single leaf node, the max latency of both the retrieval and the ranking phase showed sharp spikes every few minutes and fell back almost to normal between them. Both phases run in a single thread per request, so something was either taking that thread off the CPU or making its work slow.

Removing variables had no effect at all. Manas already had exclusive nodes and dedicated networking; the test cluster was moved to a larger EC2 instance type that held the whole index in memory, given the same AMI as the production EC2 Manas clusters, run without cgroup CPU and memory limits, and finally run directly on the node outside a container. Sampling CPU, memory and network use, comparing preemption with perf, checking cache use and kernel lock contention, and shielding the Manas leaf process onto almost all CPUs with taskset and cpusets showed no CPU stealing or preemption and did not change the spikes.

The team then paused nonessential processes one at a time with sudo kill -STOP, including logging, stats pipelines, security, the kubelet and Pinterest daemons, and the latency spikes disappeared as soon as cAdvisor was suspended. perf top on the CPUs not used by Manas showed most CPU time in memory-related kernel spans, with cAdvisor calling the kernel's smaps feature.

Symptom High latencyIntermittent timeouts

Root cause

Root cause Software defect

Component Telemetry agentLinux kernel
Contributing factor Unreviewed defaults
Trigger
–

Detection

Performance testing with traffic sent to a test cluster showed a small but persistent percentage of requests timing out.

Remediation

Action items

–

Lessons

CPU shielding does not fully isolate a process from its neighbours. Showing early that the node image was not the cause narrowed the search and avoided dead ends. Simple blackbox debugging, breaking things until the system stops working, works as a way of bisecting the problem.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Telemetry agent, Linux kernel among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.