Incident Library

IL-0439 Cybozu · 2018-06-28 (published)database

Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cache

The page cache is the kernel feature that keeps copies of storage data in memory: the first read copies data from the device into the page cache, later reads are served from memory, and writes go to the page cache and are written back to the device later. Because memory is several orders of magnitude faster than SSDs or HDDs, once the page cache is lost, reads go to the much slower storage device again. The statistics from free and sar suggested that expanding the block device invalidated the page cache of the filesystem on it.

In the kernel source, check_disk_size_change(), called during a block device size change, logs "%s: detected capacity change from %lld to %lld\n" and calls flush_disk(bdev, false) whenever the old size bdev_size differs from the new size disk_size, which invalidates the page cache (and other caches such as those used for path resolution) of the filesystem on the device, on shrinking and on expanding alike. No kernel setting avoided this. Using git blame to find the commit that added the flush_disk() call, Cybozu learned its purpose: on shrinking, data being read or written in the removed area loses its destination and causes I/O errors, and expansion could cause the same problem in the special case of expanding right after shrinking. A full reading of the related source showed that the kernel's locking makes that case impossible, so invalidation on expansion was unnecessary.

Cybozu's patch to fs/block_dev.c calls flush_disk() only when bdev_size > disk_size, that is when the device shrinks, and otherwise only sets bdev->bd_invalidated = 1 when disk_part_scan_enabled(bdev->bd_disk) is true. The patched kernel was tested in the development environment and applied in production, which ended the problem inside Cybozu. The patch was also submitted upstream as "[RFC PATCH] fs: don't flush pagecahce when expanding block device", got no reply at first, and after several simplified resubmissions over a few months was taken into the next tree and finally into Linux v4.17.

Severity
–
Scope
Database access slowed down temporarily after each filesystem expansion, showing as slower application responses for customers
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

For years, at Cybozu, database access slowed down for a while each time a filesystem was expanded. Customers saw it as slower responses from the application.

Part of Cybozu's data is stored in SQLite. When a database grows and the filesystem runs short of free space, operators first expand the block device the filesystem sits on and then expand the filesystem on that device. The slowdown came after the second step.

free and sar showed that the amount of page cache in the system dropped sharply after the block device was expanded. After the filesystem was expanded, storage device access rose in step with the drop in database performance, and fell again as performance recovered.

Symptom High latency

Root cause

Root cause Software defect

Component Linux kernelDatabase
Contributing factor –
Trigger
Operators expanded a block device, a RAID1 multiple device made of two iSCSI devices backed by LVM volumes on remote servers, and then expanded the filesystem on it because the database had grown.

Detection

The slowdown appeared as worse response times of the application customers use.

Remediation

Action items

Lessons

When the intent of kernel code is unclear, the comments and the description of the commit that introduced it are the standard place to learn why it was written. The fix was 4 added lines and 2 removed lines, but reaching it took much investigation, so the effort of a source fix does not follow the number of lines changed.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Linux kernel, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.