IL-0381 Linode · 2022-10-10database

A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts

Linode Object Storage in Newark had two related incidents, on 10 and 16 October 2022, the second while the first was still under investigation. Linode found a hardware problem on one back-end Object Storage node. Because the hardware had not failed outright, the back-end node kept going up and down, and that instability set off excessive recovery operations in the cluster. The recovery work degraded performance across the service and in some cases made the front-end gateways return 504 gateway timeouts. At 10:31 UTC on 16 October the hardware failed outright and Linode removed it from the cluster; the cluster then recovered, client IO stopped bottlenecking and the gateways served clients normally again. Linode said the incidents showed gaps in its internal monitoring.

Severity: –
Scope: Linode Object Storage in Newark: reduced performance and availability, with 504 gateway timeouts in some cases
Users affected: –
Region: Newark
Time to detect: –
Time to resolve: –

Symptoms

In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

Symptom Intermittent timeouts Throughput degradation

Root cause

Root cause Hardware failure

Component Object storage

Contributing factor Monitoring blind spot Automation amplification

Trigger: One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

Detection

–

Remediation

Removed the failed back-end infrastructure from the Object Storage cluster

Action items

Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly

Lessons

Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

Sources

https://status.linode.com/incidents/5wgmd45h7h1j · official-status · archived copy

Related cases

IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024
same root cause · shared Object storage

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Object storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0381
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0381/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.