Incident Library

IL-0578 GMO Pepabo · 2020-06-26 (published)networking

Static content behind an HAProxy load balancer in an OpenStack Octavia amphora was delayed by tens of seconds

HAProxy ran in a VM inside an amphora of the OpenStack Octavia service, and the VM had only 2 GB of RAM. The Linux kernel sets the default net.ipv4.tcp_mem at boot from the amount of memory, so the TCP memory thresholds on this VM were low. The memory the kernel had allocated to TCP socket buffers went over these thresholds, the kernel entered TCP memory pressure mode and logged TCP out of memory, and performance dropped; the later investigation concluded that this caused the delays. An engineer tuned net.ipv4.tcp_mem with sysctl, which applies immediately to all sockets without restarting processes, and the delays stopped.

The investigation could not explain at first how this state delayed the content, so the author reproduced it in a Vagrant lab and read the kernel source. Above the pressure threshold the kernel limits send and receive buffer sizes and shrinks receive windows. Above the hard limit, the third value of net.ipv4.tcp_mem, it also drops newly received segments, blocks senders until memory is available, fails connect(2) with ENOBUFS, and resets sockets that are closed with data still in the send buffer, which is when it logs TCP out of memory.

Severity
–
Scope
Delivery of static content through the HAProxy load balancer was delayed by tens of seconds
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

In a GMO Pepabo production setup, delivery of static content was delayed by tens of seconds. The static content was served by nginx on upstream hosts behind HAProxy, which ran as the HTTP load balancer.

During the response, an engineer found a host whose dmesg showed TCP: out of memory -- consider tuning tcp_mem. That host was the one running HAProxy.

Symptom High latency

Root cause

Root cause Capacity exhaustion

Component HAProxyLinux kernelCloud load balancer
Contributing factor Unreviewed defaults
Trigger
–

Detection

–

Remediation

Action items

–

Lessons

On a VM with little RAM, the boot-time default of net.ipv4.tcp_mem can be low enough to push TCP into memory pressure. TCP out of memory is not the OOM killer, and net.ipv4.tcp_mem is a system-wide limit counted in pages, unlike the per-socket net.ipv4.tcp_rmem and net.ipv4.tcp_wmem counted in bytes. The values to watch are TCP mem in /proc/net/sockstat, press in /proc/net/protocols, and TCPMemoryPressures, TCPMemoryPressuresChrono and TCPAbortOnMemory in /proc/net/netstat.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, HAProxy, Linux kernel, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.