IL-0120 OpenAI · 2025-06-09ai-infranetworking
OpenAI's cloud-hosted GPU virtual machines ran an automatic daily operating system update. On 9 June the update restarted the network management service systemd-networkd on many nodes. systemd-networkd conflicted with a networking agent OpenAI runs on its production nodes, and all routes were removed from the affected nodes, so they lost network connectivity. The lost GPU capacity caused errors of up to about 35 percent for ChatGPT and dropped API availability to 75 percent.
Engineers started re-imaging the affected VMs at 2:00 AM PDT and deployed recovery automation at 6:49 AM to restart the remaining impacted nodes. Capacity came back gradually between 8:00 AM and 3:00 PM PDT. OpenAI says there was no break-glass tooling to restore network connectivity on the affected nodes quickly, which made recovery take longer, and additional measures were needed to bring nodes back online. The incident was marked mitigated at 3:00 PM PDT on 10 June.
At 11:36 PM PDT on 9 June, alerts fired across multiple OpenAI services as a large number of cloud-hosted GPU nodes lost network connectivity, and the engineering team was escalated to immediately. Losing those nodes cut the capacity available to serve ChatGPT and the API.
By 01:45 AM PDT on 10 June errors stood at about 5 percent while capacity was rebalanced. The worst period was from 2:00 AM to 8:00 AM PDT: ChatGPT error rates peaked at about 35 percent, and API availability dropped to 75 percent, with API error rates peaking at about 25 percent.
Most major API models were fully working again by 9:40 AM PDT, the API was fully recovered at 12:30 PM, and all affected systems were restored by 3:00 PM PDT.
Symptom High error rateGPU unavailable
Root cause Breaking change in an upgrade
Alerts fired immediately across multiple services when the infrastructure began losing network connectivity at 11:36 PM PDT, and the engineering team was escalated to at once.
An unattended operating system update that restarts systemd-networkd can wipe the routes another networking agent manages, and it hits every node that takes the update at about the same time. Without break-glass tooling to restore connectivity, nodes that lost their network had to be re-imaged, which stretched recovery over many hours.
IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023same root cause · shared Node operating system image, systemd-networkd
IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025same root cause · shared Node operating system image
IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022same root cause · shared Node operating system image
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.