Incident Library
Applied filters Root causeSoftware defectComponentModel inference serverFactorNo headroomClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load
    Google Cloud2024
    Symptoms

    From 12:06 US/Pacific on 12 June 2024, services built on Vertex AI Prediction began reporting failed user requests in eight regions: us-central1, asia-southeast1, europe-west3, europe-west4, us-east1, us-west1, northamerica-northeast1 and us-east4.

    High error rateHigh latencyIntermittent timeouts
    Trigger

    Starting on 8 June 2024, a new form of user request to the GenerateContent API caused intermittent segmentation faults in the Vertex Serving Prediction API servers; on 12 June the rate of such requests increased.

    Root cause
    Software defect
    Component
    Model inference serverCloud load balancer
    Contributing factor
    Partial rolloutNo headroom
    Detection

    Internal production monitoring alerted engineers to the segfaults on 10 June 2024 at 07:35 US/Pacific, before any visible customer impact; on 12 June services built on Vertex Prediction reported failing user requests.

    Remediation
    • confirmed the new failures came from the same segfault found on 10 June
    • accelerated the rollout of the fix that was already in progress, completed at 15:59 US/Pacific
    Action items
    • Ensure early signals of server binary issues are captured in production health analysis and investigated in time
    • Ensure release validation of feature changes and updates in production
    Lessons

    A crash that only some requests trigger can stay invisible while load balancers route around the restarting servers. It becomes an outage when the share of such requests grows until too many servers are restarting at once. Here the cause was known and a fix was rolling out two days before customers were affected, but the progressive rollout had not finished when the triggering traffic increased; Google's follow-up is to investigate early signals of server binary issues in time.

    View case
Check your cluster for this

This case involves Cloud load balancer and Model inference server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Cloud load balancerModel inference server
Contributing factor
No headroomPartial rollout
Organisation
Google Cloud
  1. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load
    Google Cloud2024