Incident Library
Applied filters Root causeSoftware defectOrganisationShopifyClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0462 Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacement
    Shopify2021
    Symptoms

    In 2021 Shopify's KateSQL, a database-as-a-service platform that runs several hundred production MySQL instances on GKE, found that some MySQL pods started and shut down much more slowly than similar pods with the same data set. Instance replacement, the procedure that creates a new MySQL replica pod and stops the old one, showed pod creation times ranging from 10 to 30 minutes, which held up maintenance such as configuration changes and upgrades.

    Slow pod startup or shutdownHigh latency
    Root cause
    Software defect
    Component
    Linux kernelNode operating system imageDatabase
    Contributing factor
    Unpatched known bug
    Detection

    KateSQL's instance replacement operations showed inconsistent MySQL pod creation times, and the team then measured the mysql-init container and MySQL startup times across pods.

    Remediation
    • Cordoned older nodes, moved MySQL pods to new nodes through instance replacement and drained the old nodes
    • Forced the kubepods memory cgroup to empty its dead cgroups on an affected node to test the fix
    • Created new GKE 1.18 clusters and started creating MySQL pods on them
    Lessons

    The investigation was one of the longest the Database Platform group had carried out, because of the nature of the problem, how hard it was to reproduce and the lack of an established method to follow; once the cause was known, there were several ways to fix it.

    View case
Check your cluster for this

This case involves Database and Linux kernel. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
DatabaseLinux kernelNode operating system image
Contributing factor
Unpatched known bug
Organisation
Shopify
  1. IL-0462 Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacement
    Shopify2021