Cases1 case

1 to 1 of 1

1. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down

   OpenAI2023

   Symptoms

   At 4:53 pm Pacific on 25 January 2023 one of OpenAI's GPU clusters had a major outage, and the text-davinci-003 model went fully down.

   Service outageThroughput degradation

   Trigger

   A configuration change to the CNI (Container Network Interface) plugins, which provide network connectivity to containers, was deployed to a GPU cluster that contained a hardware variant the staging environment did not have.

   Root cause

   Configuration change

   Component

   CNI plugin

   Contributing factor

   Unrepresentative stagingMonitoring blind spot

   Detection

   Engineers immediately identified the problem as network connectivity, but it took nearly an hour to identify CNI as the cause.

   Remediation

   • put a fix for the CNI configuration change in place, which restored 80% of traffic immediately
   • deployed models back onto GPUs over about three more hours until capacity was fully stable

   Action items

   • Make staging environments contain every type of hardware configuration so they better represent production
   • Monitor and alert on the health of the CNI system directly
   • Fix the network and storage bottlenecks that limit how quickly many replicas of a model can be deployed

   Lessons

   A change that had been deployed to other clusters over 24 hours and deemed safe still broke a cluster with different hardware, so staging needs every hardware configuration that production has.

   View case

By root cause1