IL-0353 Fly.io · 2026-05-28networking
Fly.io edge proxies in US west coast regions were overloaded and flapping, with lazy loader delays of seconds
Fly.io's edges run fly-proxy, which since an earlier change loads only the Corrosion entries that active requests need through a lazy loader, and Airtime, a per-process bandwidth limit tuned to trigger near the saturation point of an edge server. On 28 May 2026 edges in the US west coast, first SJC and later LAX, were overloaded: lazy loader queries took from 500 ms to seconds, and moving Anycast traffic from SJC to LAX, and at times off the west coast, only moved the problem. Lowering Airtime's limit reduced softirq CPU and load, and bouncing the proxy, with a hard kill of all proxy processes pending shutdown, cleared the lazy loader backlog and ended the first acute phase.
The overload returned because a bug in how the proxy deployment script works with systemd had left several proxy instances running on some nodes: systemctl kill does not move a unit to a stopped state, and with Restart=always the process simply restarts. Each instance enforced Airtime alone, so the effective limit could double. Fly.io names four causes together: edge capacity in some regions had not kept up with user growth, Airtime's tuning no longer matched real traffic or other proxy scaling limits, a bug let multiple active proxy instances coexist without code for shared state, and the lazy loader queued without bound under high load. Fly.io also observed high queuing delays in inserting data into the proxy's in-memory service catalog behind a single coarse-grained lock, which it believes no longer scales with growth.
Fly.io fixed the bug that let proxy instances coexist, made Airtime apply a much stricter limit when it is not the expected active instance, and fixed load shedding in the lazy loader to bound request latency. It plans much more edge capacity, a sharded service catalog with finer-grained locking tested with Antithesis, an Airtime that reacts to overall system load, and monitoring for proxies not in their expected configuration.
- Severity
- –
- Scope
- Requests through fly-proxy on US west coast edges were delayed while the edges flapped; edges in other regions were briefly affected when traffic was moved out of the west coast
- Users affected
- –
- Region
- US west coast (SJC, LAX)
- Time to detect
- –
- Time to resolve
- –
Symptoms
On 28 May 2026, Fly.io's edges in US west coast regions began flapping, first mainly in SJC. Logs and metrics of fly-proxy, the userspace L4 and L7 load balancer on the edges, showed lazy loader latency of 500 ms to several seconds, so many new requests waited that long or longer, while proxy CPU usage and the inbound connection rate were not especially high.
Engineers first blamed one app with very large state in Corrosion, the service discovery and state propagation system the lazy loader reads from, and deployed the proxy in SJC with a temporary mitigation for it. Latency improved for a moment and then climbed sharply again once the new proxy processes warmed up. Inefficient sqlite queries were ruled out, and lock contention from higher connection rates after recent growth was considered. Airtime, the proxy's built-in defence against traffic spikes, reported higher bandwidth in SJC, but below what had been taken as the ceiling for one edge server.
Because SJC was short of edge capacity with a couple of servers out of production, Anycast traffic was shifted to LAX, which helped at first until LAX began to struggle too. At some points traffic was moved off the west coast entirely, which briefly affected edges in other regions. Lowering Airtime's bandwidth limit brought softirq CPU usage and host load average down, but lazy loader queries stayed slow until the proxy was bounced.
The problem then came back. Some affected nodes had several fly-proxy instances running at once, each with its own Airtime state that it did not share, so any Airtime limit could in effect double when heavy connections landed on another instance. The lazy loader could also build effectively endless queues waiting on sqlite connections when it was slow, and those queues cleared only when the process was bounced.
Symptom High latency
Root cause
Root cause Capacity exhaustion
- Trigger
- –
Detection
Fly.io noticed flapping in its US west coast regions, and proxy logs and metrics showed high lazy loader latency.
Remediation
- Deployed the proxy in SJC with a temporary mitigation for one app with very large state in Corrosion, which helped only briefly
- Shifted Anycast traffic from SJC to LAX
- Lowered Airtime's bandwidth limit, which reduced softirq CPU usage and host load average
- Bounced the proxy and hard-killed all proxy processes pending shutdown, which cleared the lazy loader problem
Action items
- Provision significantly more edge capacity
- Fixed the bug that let multiple proxy instances coexist
- Changed Airtime so that, for now, it applies a much stricter limit when it is not the expected active proxy instance
- Fixed load shedding in the lazy loader so request latency has a reasonable upper bound
- Shard the in-memory service catalog and move to finer-grained locking, tested with Antithesis
- Rework Airtime to react to overall system load instead of only the proxy
- Look into better monitoring for when the proxy is not in its expected configuration
Lessons
–
Sources
Related cases
IL-0341Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hoursCloudflare · 2025same root cause · shared CDN and edge network
IL-0555Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024Sentry · 2024same root cause · shared CDN and edge network
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Service discovery
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared Service discovery
IL-0275Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hoursParse.ly · 2015same root cause · shared Service discovery
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0353
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0353/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.