Incident Library

IL-0450 Toss Securities · 2026-10-19 (published)database

Oracle RAC commits behind order APIs kept waiting over 0.1 s on log file sync while average host CPU had headroom

The database is a two-node Oracle RAC on servers with 4 CPU sockets of 8 physical cores each, 32 cores in total, shown as 64 logical CPUs with hyper-threading. Each socket is its own NUMA node, and each NIC port is physically connected to one socket, so the slot a NIC sits in decides which socket handles its interrupts. The two interconnect NICs were bonded in Active-Standby mode, so in normal operation all interconnect traffic, and all its interrupts, went through the active port and the one socket it was attached to. NIC interrupts are handled before ordinary processes: when a packet from the other node arrived while LGWR ran on one of those cores, the kernel paused LGWR to handle the interrupt on the same core, the redo write was delayed, and every session waiting for its commit waited too. Each stall was very short, but in busy periods they repeated until commit waits became visibly long, even though the host as a whole had idle CPU.

The team moved the two interconnect NICs into slots connected to different sockets and switched the bonding to Active-Active so both ports carry traffic, which spreads the interrupts over two sockets without any setting to remember at startup. In the same peak period afterwards, the physical cores handling more than 1,000 interrupts per second rose from 8 to 15 (16 to 31 logical CPUs) and the Gini coefficient of interrupt concentration fell from 0.80 to 0.67. On another RAC with the same change, on a day with 28% more concurrent users, average CPU in the first 10 minutes after market open fell from 36.6% to 17.4% and the 10-minute maximum from 42.5% to 21.9%. On the original server, kernel CPU (%system) in the peak 15 minutes fell from 7.77% to 5.89% on a day with 33% higher load, and 8 of the 9 SQL statements common to the AWR reports before and after ran faster per execution, the six main business statements by 10.5% to 30.8%. The change was then applied to every Oracle DB server in operation, and the log file sync waits of 0.1 s or more mostly improved.

Severity
–
Scope
Commits on the Oracle RAC database behind the order and execution APIs repeatedly waited over 0.1 s on log file sync, delaying API responses
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

Commits on the Oracle RAC database behind Toss Securities' order and execution APIs began to wait more than 0.1 s on the log file sync wait event, again and again, and API responses were delayed by the same amount. The DBA team checked storage, the OS, the network, the application and Oracle in turn, and every average metric was in the normal range; the host CPU also had headroom.

log file sync is the time a committing session waits until the background process LGWR (log writer) has written the redo log to disk. Every commit of an instance goes through LGWR, so even a short stall of LGWR makes every session that commits in the meantime wait with it. During regular trading hours, when orders pile up, even short delays mattered.

Because the delay was intermittent, the team suspected that averages were hiding it and traced LGWR in finer detail. Recording with pidstat which CPU LGWR ran on showed a pattern: the delay appeared only when LGWR ran on CPU 28. Reading /proc/interrupts twice, 30 seconds apart, showed that NIC interrupts were handled by only 8 of the 32 physical cores (16 of 64 logical CPUs), exactly one CPU socket, and CPU 28 was one of the cores dedicated to handling those interrupts.

The team first tried to spread the interrupts with irqbalance, but on this machine the setting did not take effect because of constraints on how NIC queues map to CPUs, and the interrupts stayed on the same cores. Pinning LGWR to a CPU of another socket with taskset was rejected because it would have to be set by hand at every Oracle start and could easily be missed.

Symptom High latency

Root cause

Root cause Capacity exhaustion

Component DatabaseNIC interrupt placement and IRQ affinity
Contributing factor Monitoring blind spot
Trigger
–

Detection

The DBA team saw log file sync waits of more than 0.1 s recurring from some day on, while average metrics on every layer looked normal.

Remediation

Action items

–

Lessons

Averages hide a bottleneck on a single core: the host CPU can be mostly idle while one core is full, so for intermittent delays it is worth checking with pidstat -t -p <pid> 1 which CPU a process runs on. Two reads of /proc/interrupts at a fixed interval show which device's interrupts land on which cores. On multi-socket servers, NIC and HBA ports are tied to a NUMA node, so slot placement should spread them over sockets, and Active-Standby bonding, while a redundancy setting, also puts all normal traffic and interrupts on one port; for critical paths Active-Active with ports on different sockets is worth considering. irqbalance is not always an answer, because the hardware and driver may not allow redistribution in software. No single layer was faulty; the problem was the combination of NIC slot, bonding mode and the core LGWR was scheduled on.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, NIC interrupt placement and IRQ affinity among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.