Incident Library
Applied filters Root causeHardware failureComponentObject storageFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts
    Linode2022
    Symptoms

    In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

    Intermittent timeoutsThroughput degradation
    Trigger

    One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

    Root cause
    Hardware failure
    Component
    Object storage
    Contributing factor
    Monitoring blind spotAutomation amplification
    Remediation
    • Removed the failed back-end infrastructure from the Object Storage cluster
    Action items
    • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
    • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly
    Lessons

    Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

    View case
Check your cluster for this

This case involves Object storage. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Hardware failure1 case

Component
Object storage
Contributing factor
Automation amplificationMonitoring blind spot
Organisation
Linode
  1. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts
    Linode2022