IL-0296 Transloadit · 2016-05-30 (published)databaseaws
Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customers
Transloadit queues Jobs and passes heartbeats between Uploaders and Drones through a regional Redis cluster on AWS Elasticache. HLS splits large videos into many small fragments and can mean a thousand times more metadata for each video, and a steep rise in HLS traffic, combined with more Drones being online, saturated the network bandwidth of the 3 node us-east cluster. Jobs and heartbeats were lost and 81 customers were affected. Transloadit found no change to its own code or infrastructure that explained the sudden saturation.
As a stopgap the team re-added lost messages and relaxed heartbeat monitoring, which made Assemblies take longer. A wider investigation found that Redis was not on the latest version, which fixed several bugs including some marked as data loss fixes, so Redis was upgraded; an audit of the client library found no problems; and monitoring, metrics and alerts were extended. Once bandwidth throttling was clear, the cluster was doubled to six nodes and then grown to nine, which made it worse: every write went to the leader, which now had to replicate it to eight nodes instead of two, four times the outgoing traffic, for data that is write-heavy and deleted soon after it is written. Because missing data can be re-added and Elasticache requires replication for automatic failover, the team cut the read replicas down to one, and traffic, failure rates and Assembly times dropped immediately.
Afterwards Transloadit added logging and alarms for approaching saturation, capped how far Drones scale up in each region, and began adding sharding per Uploader on top of the existing sharding per region, because a single Redis node accepting all writes could not carry further growth.
- Severity
- –
- Scope
- Job queueing and heartbeats between Uploaders and Drones in the us-east region; slower Assemblies and some that could not be replayed
- Users affected
- 81 customers
- Region
- us-east
- Time to detect
- –
- Time to resolve
- –
Symptoms
Reports came in of more WORKER_JOB_ERROR errors, which tell Transloadit there is a communication problem between its Uploaders, the machines that receive files and orchestrate Assemblies, and its Drones, the machines that run encoding tasks. Many Assemblies became much slower because retries with exponential back-off kicked in, and a few could not be replayed.
Logs and debugging showed that Jobs queued in the us-east regional Redis cluster were being lost, along with the heartbeats between Uploaders and Drones that let a Job move to another Drone when its Drone stops sending them. Redis commands, connections and swap usage were monitored, but bandwidth was not, and saturating the network cards of a 3 node cluster had seemed unlikely; on a closer look that was exactly what was happening.
Doubling the cluster to six nodes raised bandwidth use at once, but it quickly reached new saturation points and many Assemblies stayed slow. Adding three more nodes made things worse: the cluster leader alone was sending about 4Gbit/s of outgoing traffic, more than Amazon advertises as possible, while traffic on the other Redis nodes stayed within limits, even though reads should have been spread across the whole cluster.
Root cause
Root cause Capacity exhaustion
- Trigger
- A steep increase in HLS traffic, together with more Drones online processing the same volume faster, which put more load on the central Redis nodes.
Detection
Reports of an increased number of WORKER_JOB_ERROR errors.
Remediation
- Re-added lost messages and relaxed heartbeat monitoring as a stopgap
- Upgraded Redis to the latest stable version
- Doubled the Redis cluster to six nodes and then added three more, which made it worse
- Brought the number of read replicas down to one, which cut traffic and failure rates
- Capped how far Drones scale up in each region
Action items
- Add logging and alarms for approaching saturation
- Find ways to decentralise the traffic between the Redis nodes and the Drones and Uploaders
- Add sharding per Uploader on top of the sharding per region
Lessons
To handle more traffic Transloadit had to use fewer Redis nodes, not more: each added replica multiplied the replication traffic from the single leader that takes every write, so scaling up the number of nodes was the wrong call. Bandwidth had not been among the monitored Redis metrics.
Sources
Related cases
IL-0432AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limitHello Inc. · 2026same root cause · shared In-memory cache
IL-0379InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026same root cause · shared Database replication
IL-0290GitLab.com's Redis Sidekiq cluster ran out of memory and lost quorum, causing 500 errors and a 12 million job backlogGitLab · 2026 · sev1same root cause · shared In-memory cache
IL-0288ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026same root cause · shared Database replication
IL-0305Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026same root cause · shared Database replication
IL-058129CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week saleMUSINSA · 2025same root cause · shared In-memory cache
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared In-memory cache
IL-0067Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restartBC Government · 2025 · sev1same root cause · shared In-memory cache
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared In-memory cache
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared In-memory cache
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared In-memory cache
IL-0453Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forklessHyperconnect · 2022same root cause · shared In-memory cache
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared In-memory cache
IL-0443A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019same root cause · shared Database replication
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared In-memory cache
IL-0334Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went downInstapaper · 2017same root cause · shared Database replication
IL-0520logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessionscloud.gov · 2016same root cause · shared In-memory cache
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0296
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0296/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.