Cases2 cases

1 to 2 of 2

1. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration change

   Google Cloud2024

   Symptoms

   From 22 October 2024 at 14:08 US/Pacific, customers deploying large models, those that need more than 100 GB of disk, through Vertex AI Online Prediction (Vertex Model Garden deployments) had their deployments fail with the error "Model server never became ready".

   Deploys failing

   Trigger

   A recent change introduced an internal storage provisioning configuration error.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   –

   Detection

   The incident began on 22 October 2024 at 14:08 but was first posted on the status page on 4 November 2024 at 13:17 US/Pacific, about 13 days later. The first post said large Vertex Model Garden deployments (greater than 100GB) on a GKE Autopilot cluster might fail.

   Remediation

   • rolled back the configuration change that introduced the storage provisioning error

   Lessons

   A configuration error that affects only one class of deployments, here models needing more than 100 GB of disk, can stay in place across most regions for almost two weeks. The visible symptom, a model server that never becomes ready, does not point at storage provisioning, and the fix was to roll the configuration back. The report lists no follow-up actions.

   View case
2. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out

   DFDS2020

   Symptoms

   At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

   Deploys failingControl plane lag

   Trigger

   During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

   Root cause

   Configuration change

   Component

   Admission webhookKubernetes API server

   Contributing factor

   No configuration testsUnverified assumptionMonitoring blind spot

   Detection

   Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

   Remediation

   • toggled the Alcide admission controller to verify that it caused the timeouts
   • permanently deleted the Alcide admission controller

   Action items

   • Monitor Kubernetes deployments for timeouts
   • Monitor 5xx status codes from the Kubernetes API server

   Lessons

   New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

   View case

By root cause1