Cases1 case

1 to 1 of 1

1. IL-0441 A ZOZO SQL Server database stalled and applications hit timeout storms whenever batch executions peaked in sales

   ZOZO2021

   Symptoms

   From around mid-2020, a SQL Server database behind one ZOZO feature made the applications throw many timeout errors whenever batch executions per second passed a certain threshold.

   Intermittent timeoutsDatabase overloadLock contention or deadlock

   Trigger

   Traffic increases during sales and lucky-bag launches pushed batch executions per second on the database above a threshold.

   Root cause

   unknown

   Component

   Database

   Contributing factor

   –

   Remediation

   • Fixed the plan guides of the four stored procedures whose compilation waits caused blocking
   • Disabled automatic statistics updates on the affected tables during periods of expected high traffic
   • Enabled automatic statistics updates again once traffic calmed down
   • Compressed the large indexes held in the data cache to use memory more efficiently

   Lessons

   When the root cause of a failure cannot be found, choosing among the things observed during the failure those that can be removed or reduced, and acting on them, can still create a state in which the failure no longer happens.

   View case

By root cause1