Cases1 case

1 to 1 of 1

1. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration change

   Google Cloud2024

   Symptoms

   From 22 October 2024 at 14:08 US/Pacific, customers deploying large models, those that need more than 100 GB of disk, through Vertex AI Online Prediction (Vertex Model Garden deployments) had their deployments fail with the error "Model server never became ready".

   Deploys failing

   Trigger

   A recent change introduced an internal storage provisioning configuration error.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   –

   Detection

   The incident began on 22 October 2024 at 14:08 but was first posted on the status page on 4 November 2024 at 13:17 US/Pacific, about 13 days later. The first post said large Vertex Model Garden deployments (greater than 100GB) on a GKE Autopilot cluster might fail.

   Remediation

   • rolled back the configuration change that introduced the storage provisioning error

   Lessons

   A configuration error that affects only one class of deployments, here models needing more than 100 GB of disk, can stay in place across most regions for almost two weeks. The visible symptom, a model server that never becomes ready, does not point at storage provisioning, and the fix was to roll the configuration back. The report lists no follow-up actions.

   View case

By root cause1