Incident Library

IL-0382 Linode · 2022-10-19database

Linode Object Storage in Newark and Atlanta returned 503 Slow Down to all clients under very high inbound traffic

Under extremely high inbound traffic on 19 and 20 October 2022, Linode found several problems in the Newark and Atlanta Object Storage clusters that together caused the incidents. The rate limiting at the edge of the platform did not work as intended and let in much more traffic than the advertised limit of 750 requests per second per bucket, so excessive load reached the internal Object Storage infrastructure. At peak load that infrastructure went past two cluster safety thresholds, the maximum number of concurrent requests and the maximum throughput the back-end cluster is allowed to carry, and answered with 503 - Slow Down. These back-end limits were too conservative and apply to all clients at once, so clients were rate limited too early. The internal Object Storage also used excessive memory, which Linode traced to a bug tracker item for Ceph: certain configuration tunables can cause excessive memory use during storage pressure events. Some internal components suffered from resource contention, and the front-end gateways were overloaded and short of resources.

Linode fixed the edge rate limiting so that it enforces the advertised 750 requests per second per bucket, adjusted the Ceph tunables to avoid runaway memory during high-pressure storage events such as cluster recovery after a hardware or network failure, moved the contended internal components to less busy, more stable and higher-performance infrastructure, and adjusted performance tunables on the internal components and the back-end clusters so that customers are not rate limited too early.

Severity
–
Scope
Object Storage in Newark and Atlanta: degraded performance, reduced availability and 503 - Slow Down responses for all clients; Object Storage functions in Linode Cloud Manager, including bucket listing for some customers
Users affected
–
Region
Newark and Atlanta
Time to detect
–
Time to resolve
–

Symptoms

On 19 and 20 October 2022, Linode Object Storage in Newark and Atlanta slowed down, became less available and rate limited many requests with "503 - Slow Down" HTTP responses, after the clusters received extremely high inbound traffic in both request count and bandwidth.

Once the back-end rate limits started, they hit every client, including clients that had not exceeded the advertised limits. Internal Linode systems that use Object Storage were also rate limited from time to time, which affected Object Storage functions in the Linode Cloud Manager, and some customers could not list their buckets there. The internal Object Storage infrastructure also used excessive memory, and resource contention on some internal components made performance inconsistent.

Symptom High error rateMemory pressure

Root cause

Root cause Capacity exhaustion

Component Object storage
Contributing factor Unpatched known bugExcessive blast radius
Trigger
Extremely high inbound traffic, in both total requests and throughput, reached the Newark and Atlanta Object Storage clusters on 19 and 20 October 2022.

Detection

–

Remediation

Action items

Lessons

Linode's per-bucket rate limiting at the edge is meant to keep excess load off the back end; when it works as intended, the back-end limits, which throttle every client alike, should never come into play.

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Object storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.