Cases12 cases

6 to 10 of 12

1. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes

   Google Cloud2022

   Symptoms

   From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.

   High error rateResources deletedService outage

   Trigger

   A faulty resource cleanup process marked Vertex Prediction Endpoints as deleted, globally.

   Root cause

   Software defect

   Component

   Model request routing

   Contributing factor

   Automation amplificationExcessive blast radius

   Detection

   The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.

   Remediation

   • restored the Vertex Prediction Endpoints that had been marked as deleted

   View case
2. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404

   DFDS2021

   Symptoms

   At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.

   Service outageHigh error rateProbe failures

   Trigger

   The automated Kubernetes node rollover on 2021-09-06 restarted all the Traefik v2 pods, five days after a malformed IngressRoute manifest had been applied.

   Root cause

   Software defect

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Change with a delayed effectNo configuration testsExcessive blast radius

   Detection

   All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.

   Remediation

   • changed the ALB health check path, which made the targets healthy but did not stop the 404s
   • rolled back by hand to Traefik v1 as the main ingress controller
   • deleted the malformed IngressRoute by hand and confirmed the Traefik v2 bug
   • applied the changes upstream

   Action items

   • Upgrade Traefik to v2.5, where the bug is fixed, and verify that it removes the bug
   • Change the Traefik v2 health check path to something more robust
   • Set up an external liveness probe on the ingress controller
   • Consider workload segmentation to limit the blast zone
   • Look into monitoring the Traefik v2 API for rapid changes
   • Look into smoke testing during and after node rollover

   Lessons

   A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik. On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.

   View case
3. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data

   Palark2020

   Symptoms

   Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.

   Data lossService outage

   Trigger

   A RedisFailover change that raised the number of Redis replicas from one to two and, in the same rollout, lowered the containers' memory requests and limits from 22 GB to 19 GB.

   Root cause

   Software defect

   Component

   Application databasePersistent volumes and stateful setsContainer memory limit

   Contributing factor

   No staged rolloutSingle point of failureUnverified assumption

   Remediation

   • restored the Redis data from the backup prepared before the rollout

   Action items

   • Report the readiness probe flaw to the Redis Operator project and test the developers' fix
   • Reconsider the use of the Spotahome Redis Operator
   • Clean up the Redis databases and fix the application code that created the excess keys
   • Consider Redis built-in clustering and sharding

   Lessons

   Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty. A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.

   View case
4. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files

   thredUP2018

   Symptoms

   While thredUP's move to Kubernetes seemed to be going well, Datadog showed the number of open files on the nodes climbing steadily, with no obvious reason.

   Service outageFile descriptor exhaustion

   Trigger

   Containers starting on a node, which in Kubernetes happens constantly with every deploy and every pod that moves between nodes.

   Root cause

   Software defect

   Component

   Telemetry agent

   Contributing factor

   Unverified assumption

   Detection

   Datadog showed the number of open files on the nodes rising.

   Remediation

   • killed the affected node, brought up a new one and redeployed

   Action items

   • Move away from the log aggregation vendor

   Lessons

   Libraries and agents that worked with the EC2 deployment lifecycle may not cope with Kubernetes, where containers start, stop and move between nodes constantly. Each one needs to be checked against the new lifecycle instead of being trusted because it worked before.

   View case
5. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures

   AirMap2018

   Symptoms

   At about 15:00 UTC on 19 July 2018, monitoring alerted on rising 5xx responses from several APIs, most notably pilot and tiles. Logins failed first, because the pilot service fetches the user's profile, and from there all services became unresponsive. Every affected API ran on the production Kubernetes cluster.

   High error rateNode NotReadyDNS resolution failuresContainer crash loopService outage

   Root cause

   Software defect

   Component

   kubelet

   Contributing factor

   Unpatched known bug

   Detection

   Monitoring alerted on an increase in 5xx responses from several APIs, most notably pilot and tiles.

   Remediation

   • restarted the kubelets on the stuck worker nodes, which did not last
   • restarted all worker nodes in a staggered order starting at about 18:00 UTC, which took about 1.5 hours

   Action items

   • Restart nodes on a rolling schedule to keep nodes and kubelets healthy until the fixes land upstream and reach the cloud provider, Microsoft Azure

   Lessons

   When nodes keep falling back to NotReady after kubelet restarts and pods fail on network and DNS errors, a known upstream bug may be the cause; matching the symptoms to the upstream issue tracker, with help from the community, found the cause and the fix candidates.

   View case

By root cause1