Cases52 cases

31 to 35 of 52

1. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours

   Jenkins project2021

   Symptoms

   At 11:32 UTC on 18 November 2021, a user reported in the jenkinsci/jenkins Gitter channel that https://www.jenkins.io/doc/pipeline/steps/ was answering HTTP/503 errors with the message "Guru meditation" from Fastly's varnish.

   Service outageResources deleted

   Trigger

   The ingress-nginx Helm chart was upgraded from 4.0.6 to 4.0.8 at 10:42 UTC, and the new chart added an admission webhook rule that forbids Nginx configuration blocks in Ingress annotations.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerAdmission webhookHelmCDN and edge network

   Contributing factor

   Unverified assumptionCache-masked failureMonitoring blind spotHarmful manual action

   Detection

   A user reported in the jenkinsci/jenkins Gitter channel at 11:32 UTC that a jenkins.io page answered HTTP/503 errors, and status.jenkins.io reported the incident at 11:49 UTC.

   Remediation

   • Disabled the automated systems, rolled the ingress-nginx chart back to 4.0.6 and deleted the admission webhook by hand
   • Removed the forbidden annotations from the jenkins.io configuration
   • Upgraded the jenkins.io Helm chart to the last version known to work with ingress-nginx 4.0.8
   • Redeployed the jenkins.io Helm chart through the automated system and told Fastly to drop all cached pages

   Action items

   • Always read the changelog in depth when updating the ingress controllers, with two people pairing in real time
   • Add custom Datadog monitoring for www.origin.jenkins.io so that such an outage is detected
   • Make the monitor send the Host header www.jenkins.io so it is not redirected to Fastly
   • Check a set of pages, such as the pipeline steps page, not only the front page
   • Consider an ephemeral k3s end-to-end test before deploying to production

   Lessons

   Trusting a patch version of a Helm chart for the Ingress controller without reading its changelog in depth let a new admission rule remove the jenkins.io Ingress in production. Without monitoring on origin.jenkins.io, a user found the outage. A complete staging cluster with the same DNS setup would have caught the problem before production.

   View case
2. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour

   DFDS2021

   Symptoms

   After the switch to Traefik v2, functional tests indicated that everything was fine, and this was reported on Slack at 08:44 UTC. The first sign of trouble came at 09:26 UTC, when a developer reported a high count of HTTP 307 redirects, especially on the Okta service. At 09:32 UTC the team learned that the online booking system was down.

   Service outage

   Trigger

   At 08:22 UTC the AWS load balancers were updated to send requests to the new Traefik v2 reverse proxy as part of the migration away from Traefik v1.

   Root cause

   Configuration change

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Missed in reviewMonitoring blind spot

   Detection

   A developer reported a high count of HTTP 307 redirects, in particular on the Okta service, at 09:26 UTC, 64 minutes after the change; six minutes later the team was told the online booking system was down. The functional tests run after the switch had not caught it.

   Remediation

   • Rolled back the switch to Traefik v2, putting rollback ahead of investigation because of how critical the booking system is

   Action items

   • Consider how critical announcements can be made more visible
   • Consider whether the ALBs used for testing should have been shared with the teams so they could test their own scenarios

   Lessons

   Cloud Engineering cannot test every application scenario on its own, even after testing its own applications for weeks. A header that changes value behind a new ingress controller, here x-forwarded-proto going from https to http, can break an authentication flow while basic functional tests still pass. DFDS asks whether diffing request headers before and after the migration would have shown the problem in advance.

   View case
3. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1

   GitLab2021

   Symptoms

   At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

   Scale-up failureHigh latencyPods PendingDeploys failing

   Trigger

   Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

   Root cause

   Cloud provider outage

   Component

   Cluster autoscalerNode pool

   Contributing factor

   Monitoring blind spotLimited debug accessOverlapping failures

   Detection

   There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

   Remediation

   • added a node pool that uses pd-standard disks
   • scaled the new node pool by hand across all three zones to relieve the API
   • brought more pd-standard node pools online for the other blocked workloads
   • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
   • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
   • removed the SSD-backed node pools after a day without problems

   Action items

   • Move Kubernetes services to pd-standard disks (delivery#1837)
   • Capture resource exhaustion errors as a metric

   Lessons

   A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

   View case
4. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired

   Jenkins project2021

   Symptoms

   From 18:02 UTC on 2 June 2021, the Jenkins infrastructure AKS cluster logged errors about pods that were unable to mount volumes and about scaled-up nodes that were unable to obtain a network virtual interface from the CNI.

   Credentials unavailablePods stuck in ContainerCreating

   Trigger

   The Azure Service Principal credential of the AKS cluster expired at 17:53 UTC on 2 June 2021.

   Root cause

   Certificate or credential expiry

   Component

   Cluster cloud identityPersistent volumes and stateful setsCNI pluginNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   The error behind the failures was found in the AKS UI console while the team investigated the root cause of the LTS 2.289.1 release problem.

   Remediation

   • Renewed the AKS cluster's Azure Service Principal credential, which made AKS restart all nodes automatically

   Action items

   • Add an automatic PagerDuty check that alerts a week before the service principal expires, as is done for certificates
   • Keep a team-level calendar for manual operations such as resetting a credential
   • Consider managed identities, which renew automatically, instead of the service principal

   View case
5. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com

   CircleCI2021

   Symptoms

   At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.

   Service outageResources deleted

   Trigger

   An update deployed to a test Kubernetes cluster also removed Traefik and Calico resources from CircleCI's primary production cluster.

   Root cause

   Faulty deployment

   Component

   Ingress controllerCalico

   Contributing factor

   Excessive blast radiusNo runbookMonitoring blind spot

   Detection

   CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.

   Remediation

   • restored Traefik in the production cluster
   • restored Calico in the production cluster, after which pods served customer requests again

   Action items

   • Improve network-level documentation and runbooks to cover every layer of the Kubernetes networks
   • Improve monitoring and alerting for the network infrastructure
   • Add guardrails that isolate the production and staging clusters so they cannot interfere with each other

   Lessons

   A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again. Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.

   View case

By root cause9