Cases1 case

1 to 1 of 1

1. IL-0439 Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cache

   Cybozu2018

   Symptoms

   For years, at Cybozu, database access slowed down for a while each time a filesystem was expanded. Customers saw it as slower responses from the application.

   High latency

   Trigger

   Operators expanded a block device, a RAID1 multiple device made of two iSCSI devices backed by LVM volumes on remote servers, and then expanded the filesystem on it because the database had grown.

   Root cause

   Software defect

   Component

   Linux kernelDatabase

   Contributing factor

   –

   Detection

   The slowdown appeared as worse response times of the application customers use.

   Remediation

   • Patched check_disk_size_change() so the kernel invalidates the page cache only when a block device shrinks
   • Tested the patched kernel in the development environment and applied it in production

   Action items

   • Get the patch merged into the upstream Linux kernel so it need not be backported at each future OS upgrade

   Lessons

   When the intent of kernel code is unclear, the comments and the description of the commit that introduced it are the standard place to learn why it was written. The fix was 4 added lines and 2 removed lines, but reaching it took much investigation, so the effort of a source fix does not follow the number of lines changed.

   View case

By root cause1