Incident Library
Applied filters Root causeBreaking change in an upgradeComponentsystemd-networkdSymptomGPU unavailableClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API
    OpenAI2025
    Symptoms

    At 11:36 PM PDT on 9 June, alerts fired across multiple OpenAI services as a large number of cloud-hosted GPU nodes lost network connectivity, and the engineering team was escalated to immediately. Losing those nodes cut the capacity available to serve ChatGPT and the API.

    High error rateGPU unavailable
    Trigger

    A routine daily scheduled update of the host operating system on the cloud-hosted GPU servers restarted systemd-networkd on the affected nodes.

    Root cause
    Breaking change in an upgrade
    Component
    systemd-networkdNode operating system image
    Contributing factor
    No staged rolloutLimited debug access
    Detection

    Alerts fired immediately across multiple services when the infrastructure began losing network connectivity at 11:36 PM PDT, and the engineering team was escalated to at once.

    Remediation
    • re-imaged the affected GPU virtual machines
    • deployed recovery automation to restart the remaining impacted nodes
    • halted the background update mechanisms on the GPU VMs
    Action items
    • Disable automatic daily updates on GPU VMs so updates run in a controlled, scheduled way
    • Update system configurations to prevent conflicts between systemd-networkd and the networking service
    • Audit VM configurations across the fleet to find and reduce similar risks
    • Improve recovery speed for critical infrastructure such as GPU VMs and clusters
    • Run regular disaster recovery drills
    Lessons

    An unattended operating system update that restarts systemd-networkd can wipe the routes another networking agent manages, and it hits every node that takes the update at about the same time. Without break-glass tooling to restore connectivity, nodes that lost their network had to be re-imaged, which stretched recovery over many hours.

    View case
Check your cluster for this

This case involves Node operating system image and systemd-networkd. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
Node operating system imagesystemd-networkd
Contributing factor
Limited debug accessNo staged rollout
Organisation
OpenAI
  1. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API
    OpenAI2025