Cases11 cases

4 to 8 of 11

1. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration change

   Google Cloud2024

   Symptoms

   From 22 October 2024 at 14:08 US/Pacific, customers deploying large models, those that need more than 100 GB of disk, through Vertex AI Online Prediction (Vertex Model Garden deployments) had their deployments fail with the error "Model server never became ready".

   Deploys failing

   Trigger

   A recent change introduced an internal storage provisioning configuration error.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   –

   Detection

   The incident began on 22 October 2024 at 14:08 but was first posted on the status page on 4 November 2024 at 13:17 US/Pacific, about 13 days later. The first post said large Vertex Model Garden deployments (greater than 100GB) on a GKE Autopilot cluster might fail.

   Remediation

   • rolled back the configuration change that introduced the storage provisioning error

   Lessons

   A configuration error that affects only one class of deployments, here models needing more than 100 GB of disk, can stay in place across most regions for almost two weeks. The visible symptom, a model server that never becomes ready, does not point at storage provisioning, and the fix was to roll the configuration back. The report lists no follow-up actions.

   View case
2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes

   DFDS2022

   Symptoms

   At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

   Deploys failingContainer crash loop

   Trigger

   A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

   Root cause

   Certificate or credential expiry

   Component

   Policy engineAdmission webhook

   Contributing factor

   Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal

   Detection

   An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

   Remediation

   • reproduced the issue in a sandbox cluster and deployed the fix there first
   • updated Kyverno to a newer Helm chart version in the affected production cluster

   Action items

   • Implement monitoring on Kyverno
   • Create internal documentation on the Kyverno setup
   • Investigate the exact certificate issue in the third-party application
   • Consider changing the failurePolicy on ClusterPolicy resources going forward

   Lessons

   Operational issues should be added to the 'Ops Day to Day' board from the start.

   View case
3. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1

   GitLab2021

   Symptoms

   At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

   Scale-up failureHigh latencyPods PendingDeploys failing

   Trigger

   Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

   Root cause

   Cloud provider outage

   Component

   Cluster autoscalerNode pool

   Contributing factor

   Monitoring blind spotLimited debug accessOverlapping failures

   Detection

   There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

   Remediation

   • added a node pool that uses pd-standard disks
   • scaled the new node pool by hand across all three zones to relieve the API
   • brought more pd-standard node pools online for the other blocked workloads
   • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
   • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
   • removed the SSD-backed node pools after a day without problems

   Action items

   • Move Kubernetes services to pd-standard disks (delivery#1837)
   • Capture resource exhaustion errors as a metric

   Lessons

   A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

   View case
4. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change

   UK Ministry of Justice2021

   Symptoms

   At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

   Deploys failing

   Trigger

   The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

   Root cause

   Breaking change in an upgrade

   Component

   Admission webhookIngress controller

   Contributing factor

   Unrepresentative staging

   Detection

   A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

   Remediation

   • rolled back the ModSec ingress controller module to v0.0.7

   Action items

   • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
   • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
   • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
   • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)

   View case
5. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired

   dbhang2021

   Symptoms

   KubeSpawner, which starts JupyterHub user containers on Kubernetes, failed to create containers with an error the author had never seen, saying an SSL certificate had expired.

   Control plane unresponsiveDeploys failing

   Root cause

   Certificate or credential expiry

   Component

   Kubernetes API serverkubelet

   Contributing factor

   Unknown limitNo alert on limit

   Detection

   KubeSpawner failed to create a container and logged an SSL certificate verification error.

   Remediation

   • checked certificate expiry with kubeadm alpha certs check-expiration
   • moved the old certificates and /etc/kubernetes/*.conf files to a backup directory
   • regenerated all certificates with kubeadm init phase certs all
   • regenerated the kubeconfig files with kubeadm init phase kubeconfig all
   • copied /etc/kubernetes/admin.conf to ~/.kube/config and restarted kubelet
   • restarted the kube-system containers with docker restart so they picked up the new credentials

   Lessons

   kubeadm clusters run on certificates that expire, and when the API server certificate expires kubectl cannot be used to fix it, so renewal has to happen on the control plane node with kubeadm. After renewal, the kubeconfig files and the running control plane containers must also pick up the new certificates, or kubectl returns Unauthorized.

   View case

By root cause7