Cases11 cases

8 to 11 of 11

1. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors

   GitLab2022

   Symptoms

   At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.

   High error rateHigh latencyIntermittent timeouts

   Trigger

   A k8s-workloads/gitlab-com pipeline deployed merge request !1643, a GitLab chart bump that changed the port names of Kubernetes Services and Deployments (related to the Prometheus service and pod monitors), starting with the canary environment gprd-cny at 07:49 UTC.

   Root cause

   Configuration change

   Component

   HelmCloud load balancerHAProxy

   Contributing factor

   Automation amplification

   Detection

   A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.

   Remediation

   • drained the canary environment and re-enabled it after twenty minutes
   • cancelled the deployment jobs for clusters c and d before they started
   • drained gprd-us-east1-d in HAProxy and re-enabled it five minutes later
   • raised the minimum replica count above 100 (merge request !1650)
   • drained gprd-us-east1-c in HAProxy, deployed the change to it, and re-enabled it
   • reverted the minimum replica increase (merge request !1652)

   Action items

   • Implement graceful draining of a whole regional cluster
   • Create a Chatops command that puts PagerDuty into maintenance mode

   Lessons

   Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes. GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.

   View case
2. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries

   Preply2020

   Symptoms

   Prometheus detected a high rate of 5xx responses on three services, called A, B and C in the write-up, and paged the on-call engineers through VictorOps at 22:18 EET. The 5xx rate was also visible in Kibana.

   DNS resolution failuresHigh error rateStale endpoint routing

   Trigger

   Because load in the cluster was low, the CoreDNS autoscaler reduced the number of CoreDNS pods from 3 to 2.

   Root cause

   DNS or network fault

   Component

   CoreDNSkube-proxyconntrack table

   Contributing factor

   Automation amplificationIgnored early signalNo caching

   Detection

   Prometheus alerted on a high 5xx rate on services A, B and C and paged on-call through VictorOps.

   Remediation

   • reverted the latest release of the affected service
   • a regular deploy created new nodes, the CoreDNS autoscaler added CoreDNS pods, and the conntrack table was rewritten

   Action items

   • Disable the autoscaler for CoreDNS
   • Run a DNS cache server in production
   • Monitor conntrack usage

   Lessons

   Scaling CoreDNS down is not free: if kube-proxy fails to clear conntrack entries for the removed pod, clients keep sending DNS queries to it. Known DNS weaknesses in a cluster should be prioritised before they cause an outage, and fixes that only treat consequences leave the underlying conntrack problem in place.

   View case
3. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down

   Grafana Labs2019

   Symptoms

   On Friday 19 July 2019 the Grafana Cloud Hosted Prometheus service, which is built on the Cortex project, was down for 26 minutes. New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start.

   Pod evictionHigh error rateService outageOOMKilled

   Trigger

   An engineer created a new dedicated Cortex cluster for a large customer. Its configuration did not include the new pod priorities, so all its pods got the default medium priority, and the Kubernetes cluster did not have enough free resources to fit it.

   Root cause

   Configuration change

   Component

   Pod priority and preemption

   Contributing factor

   Partial rolloutAutomation amplificationMissed in reviewNo headroomRetry storm

   Detection

   New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start of the outage.

   Remediation

   • Scaled up the underlying Kubernetes cluster to fit both the new and the production Cortex clusters
   • Diagnosed and fixed the out-of-memory errors in the authenticating reverse proxies in front of Cortex

   Action items

   • Give the production ingesters the high priority class, which should have been done before medium became the default
   • Have more people review changes to configuration that is global to the cluster
   • Require a design document for every configuration change outside a namespace
   • Automate the sizing of the frontend authenticating reverse proxy so it does not run out of memory under overload
   • Investigate the default backoff and scaling behaviour of Prometheus to prevent a stampeding herd of retries

   Lessons

   Medium should not have become the default priority until the production ingesters had the high priority, and that designation should have reached production sooner. The change was considered too minor for a design document and was only discussed in a GitHub issue; cluster-global configuration needs more scrutiny. On the positive side, once Cortex had enough capacity it recovered by itself, and Grafana Loki helped confirm that the ingesters exited gracefully.

   View case
4. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down

   Moonlight2019

   Symptoms

   On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.

   High error rateIntermittent connection failuresNode NotReadyService outageSustained high node CPU

   Trigger

   The Kubernetes scheduler put several CPU-heavy pods on the same node, including all three replicas of the website and the server-side rendering pods.

   Root cause

   Capacity exhaustion

   Component

   Pod scheduling and placementLinux kernelApplication database

   Contributing factor

   Single point of failureUnverified assumptionAutomation amplification

   Detection

   API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.

   Remediation

   • added anti-affinity rules to all major deployments so their pods spread across nodes

   Action items

   • Keep working with Google Cloud support to find and fix the cause of the node kernel panics

   Lessons

   Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.

   View case

By root cause6