Cases1 case

1 to 1 of 1

1. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load

   Google Cloud2024

   Symptoms

   From 12:06 US/Pacific on 12 June 2024, services built on Vertex AI Prediction began reporting failed user requests in eight regions: us-central1, asia-southeast1, europe-west3, europe-west4, us-east1, us-west1, northamerica-northeast1 and us-east4.

   High error rateHigh latencyIntermittent timeouts

   Trigger

   Starting on 8 June 2024, a new form of user request to the GenerateContent API caused intermittent segmentation faults in the Vertex Serving Prediction API servers; on 12 June the rate of such requests increased.

   Root cause

   Software defect

   Component

   Model inference serverCloud load balancer

   Contributing factor

   Partial rolloutNo headroom

   Detection

   Internal production monitoring alerted engineers to the segfaults on 10 June 2024 at 07:35 US/Pacific, before any visible customer impact; on 12 June services built on Vertex Prediction reported failing user requests.

   Remediation

   • confirmed the new failures came from the same segfault found on 10 June
   • accelerated the rollout of the fix that was already in progress, completed at 15:59 US/Pacific

   Action items

   • Ensure early signals of server binary issues are captured in production health analysis and investigated in time
   • Ensure release validation of feature changes and updates in production

   Lessons

   A crash that only some requests trigger can stay invisible while load balancers route around the restarting servers. It becomes an outage when the share of such requests grows until too many servers are restarting at once. Here the cause was known and a fix was rolling out two days before customers were affected, but the progressive rollout had not finished when the triggering traffic increased; Google's follow-up is to investigate early signals of server binary issues in time.

   View case

By root cause1