IL-0462 Shopify · 2021-10-21 (published)kubernetesgcpdatabase
Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacement
KateSQL runs MySQL as Kubernetes pods on GKE across several Google Cloud regions. Earlier in 2021 some MySQL pods began taking far longer than others to start and stop, slowing the instance replacement procedure. As a first mitigation the team cordoned older nodes, replaced MySQL instances so their pods moved to new nodes added by the GKE autoscaler, and drained the old nodes, which improved performance on the new pods.
A deeper analysis traced the slowness to memory allocation during InnoDB buffer pool initialisation, which spent its time in the kernel's mem_cgroup_commit_charge. Affected nodes had about 50K memory cgroups, which pointed to a cgroup leak: charging each page may have to traverse many cgroups. Bug reports, an lwn.net article and a Linux kernel mailing list thread matched the problem, and Google's kernel engineers confirmed it was another side effect of reparenting slab memory on cgroup removal in the Linux kernel memory cgroup controller. Writing 1 to /sys/fs/cgroup/memory/kubepods/memory.force_empty on an affected node brought memory cgroups down to about 1800, and a restarted MySQL pod then initialised an 80G InnoDB buffer pool in five seconds.
Shopify and Google weighed several workarounds: rebooting or cordoning nodes found by monitoring /proc/cgroups, a cron job that drops slab and page caches, moving short-lived pods to a dedicated node pool, a memory.force_empty call in the preStop hook of short-lived pods, and upgrading to COS 85, which has the upstream fixes, with the move from GKE 1.16 to 1.18 expected to bring Linux kernel 5.4 with the relevant fixes. Since a GKE upgrade was due, the team created new GKE 1.18 clusters and started creating MySQL pods there, and after several weeks buffer pool initialisation and query performance stayed consistent.
- Severity
- –
- Scope
- KateSQL instance replacement slowed: MySQL pod creation took 10 to 30 minutes, and affected pods also served slower queries
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
In 2021 Shopify's KateSQL, a database-as-a-service platform that runs several hundred production MySQL instances on GKE, found that some MySQL pods started and shut down much more slowly than similar pods with the same data set. Instance replacement, the procedure that creates a new MySQL replica pod and stops the old one, showed pod creation times ranging from 10 to 30 minutes, which held up maintenance such as configuration changes and upgrades.
Two pods with the same CPU, memory, storage and data set differed widely: katesql-n4sx0 took 2120 seconds to initialise and 1104 seconds to start, while katesql-jxijq took 74 and 17 seconds. The slow pods also became gradually slower at queries, with more slow queries among those that used temporary memory tables.
Nodes that had been running for a few months performed worse than newly provisioned ones, although kubelet, kernel and Container-Optimized OS versions were the same. Storage was ruled out. During InnoDB buffer pool initialisation mysqld used about 100% of one CPU, strace showed each mmap() call taking around 100 ms to allocate chunks of about 128MB, and a perf flame graph showed the jemalloc allocator spending most of its time in the kernel function mem_cgroup_commit_charge.
slabtop showed a dentry cache of around 20GB on a node with about 64 pods, and dropping it with echo 2 > /proc/sys/vm/drop_caches brought an immediate improvement like a node restart, although the team confirmed that its case was not the one described in a sysdig post it had followed. An affected node had approximately 50K memory cgroups, far more than a good node.
Root cause
Root cause Software defect
- Trigger
- –
Detection
KateSQL's instance replacement operations showed inconsistent MySQL pod creation times, and the team then measured the mysql-init container and MySQL startup times across pods.
Remediation
- Cordoned older nodes, moved MySQL pods to new nodes through instance replacement and drained the old nodes
- Forced the kubepods memory cgroup to empty its dead cgroups on an affected node to test the fix
- Created new GKE 1.18 clusters and started creating MySQL pods on them
Action items
–
Lessons
The investigation was one of the longest the Database Platform group had carried out, because of the nature of the problem, how hard it was to reproduce and the lack of an established method to follow; once the cause was known, there were several ways to fix it.
Sources
Related cases
IL-0112The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared Node operating system image, Linux kernel
IL-0029Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020same root cause · shared Node operating system image, Linux kernel
IL-0143GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019same root cause · shared Node operating system image, Linux kernel
IL-0439Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cacheCybozu · 2018same root cause · shared Linux kernel, Database
IL-0451Setting indexes INVISIBLE on Aurora MySQL 3.10 spiked CPU and ReadIOPS on all three readers but not on the writerDaangn · 2026same root cause · shared Database
IL-0313Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy poolDecagon · 2026same root cause · shared Database
IL-0281PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026same root cause · shared Database
IL-0310An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactionsHightouch · 2025same root cause · shared Database
IL-0150A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Database
IL-0171Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025same root cause · shared Node operating system image
IL-0436A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connectionsfreee · 2025same root cause · shared Database
IL-0147AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft Azure · 2025same root cause · shared Node operating system image
IL-0247One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to KubernetesPinterest · 2025same root cause · shared Linux kernel
IL-0527OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025same root cause · shared Database
IL-0329A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outageincident.io · 2025same root cause · shared Database
IL-0138A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025same root cause · shared Linux kernel
IL-0347About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024same root cause · shared Database
IL-0373Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four daysHeroku · 2024same root cause · shared Database
IL-0024Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024same root cause · shared Database
IL-0217Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb · 2023same root cause · shared Database
IL-0330Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 secondsincident.io · 2023same root cause · shared Database
IL-0385GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recoveringGitHub · 2021same root cause · shared Database
IL-0146New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNIMicrosoft Azure · 2020same root cause · shared Node operating system image
IL-0196Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Database
IL-0012CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020same root cause · shared Linux kernel
IL-0486Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailableFigma · 2020same root cause · shared Database
IL-0241A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019same root cause · shared Database
IL-0277Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachableElastic · 2019same root cause · shared Linux kernel
IL-0388A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017same root cause · shared Database
IL-0454Baemin billing servers could not open new MariaDB connections at the start of a card-discount eventWoowa Brothers · 2016same root cause · shared Database
IL-0536Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the clusterAmazon Web Services · 2014same root cause · shared Database
IL-0485Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slowDropbox · 2014same root cause · shared Database
IL-0589Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013same root cause · shared Database
IL-0303GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0462
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0462/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.