Cases36 cases

29 to 33 of 36

1. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes

   Grafana Labs2020

   Symptoms

   About a minute after the etcd leader changed, two replicas of the Cortex distributor started logging "context deadline exceeded" when they tried to read keys from etcd. Writes from some customers that landed on those two replicas failed, while the other distributor replicas kept working, so the service saw an error rate of about 20% on writes to the London region of Hosted Prometheus.

   High error rateIntermittent timeouts

   Trigger

   A scheduled Kubernetes version upgrade abruptly terminated the node that ran the etcd leader at 11:19:00 UTC.

   Root cause

   Software defect

   Component

   etcdNode pool

   Contributing factor

   Unreviewed defaultsNo safeguard on destructive change

   Detection

   An SLO-based alert paged the on-call engineer at 11:31, 11 minutes after the errors started. At a 20% error rate the monthly SLA would have taken about 18 hours to breach.

   Remediation

   • Restarted the two affected Cortex distributor replicas at 11:32, which stopped the errors

   Action items

   • Enable gRPC keepalive probes in the Cortex etcd client (DialKeepAliveTime 10s, DialKeepAliveTimeout, PermitWithoutStream)
   • Make the Kubernetes upgrade process terminate pods and machines gracefully, including the etcd pods managed by the etcd operator

   Lessons

   When a node dies abruptly, clients holding TCP connections to it are not told; without keepalive probes an etcd client can keep sending requests into a dead connection long after a new leader exists. A failover that works on the server side still needs clients that detect the dead peer. Node replacement during cluster upgrades should drain pods gracefully, including pods owned by operators that the upgrade scripts do not know about, and should give instances enough time to shut down.

   View case
2. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets

   loveholidays2020

   Symptoms

   Shortly after a round of changes, loveholidays saw connectivity problems and raised error rates on its aggregation service. The service is a proxy between two other services of more than 100 instances each, handles over 10,000 requests per second continuously and can spike to four times that.

   Connection resetsHigh error rateIntermittent connection failures

   Trigger

   Three changes together: a change in service architecture, halving the RAM of the nodes that run the aggregation service, and sharding a downstream service, which doubled the number of connections to track.

   Root cause

   Capacity exhaustion

   Component

   conntrack tablekube-proxyNode poolHAProxy

   Contributing factor

   Unknown limitNo capacity planningNo alert on limit

   Detection

   Rising error rates and climbing connection reset errors in HAProxy, then node_exporter conntrack metrics compared with their limit.

   Remediation

   • moved the aggregation service to nodes with more RAM
   • limited the number of aggregation service pods per node

   Action items

   • Alert when conntrack table use goes above 75%, with (node_nf_conntrack_entries / on (pod) node_nf_conntrack_entries_limit / on (pod) group_right kube_pod_info) > 0.75

   Lessons

   The conntrack limit is tied to node memory, so resizing a node pool silently resizes the connection table, and on GKE the ratio did not match the commonly cited formula. A high-traffic proxy on small nodes can exhaust it, and the symptoms are connection resets and timeouts, so conntrack use needs its own metric and alert.

   View case
3. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing

   loveholidays2020

   Symptoms

   A colleague wrote in Slack that his application was taking a long time to deploy. A deployment that normally finishes in minutes had been running for about an hour: about half of the new pods were running and serving traffic, and the other half were Pending.

   Pods PendingIP address exhaustionScale-up failure

   Trigger

   The cluster reaching 256 nodes, the most its /16 pod secondary range could hold at 256 addresses per node; every further node creation failed.

   Root cause

   Capacity exhaustion

   Component

   VPC subnet or pod address rangeCluster autoscalerNode pool

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   A colleague reported in Slack that a deployment was slow; the pending pods' events and the cluster autoscaler status ConfigMap showed the exhausted IP space.

   Remediation

   • lowered the maximum pods per node on selected node pools, cutting IP consumption by about 30%
   • moved some node pools to larger instance types

   Action items

   • Lower the maximum pods per node on node pools that do not run many pods
   • Use larger instance types in some node pools

   Lessons

   In a VPC-native GKE cluster the number of nodes, not the number of pods, uses up the pod range: by default each node reserves 256 addresses, so even a /16 is full at 256 nodes. Maximum pods per node and node size are the practical levers; larger nodes pack better but take more pods down when they fail.

   View case
4. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy

   Microsoft Azure2019

   Symptoms

   From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: "The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct." The issue title calls the error PlatformImageNotFound.

   Upgrade stall

   Trigger

   A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.

   Root cause

   Faulty deployment

   Component

   Node operating system imageNode pool

   Contributing factor

   –

   Detection

   Customers received failure messages when they tried to upgrade their clusters.

   Remediation

   • published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again
   • mitigated affected clusters by hand (AKS and VMSS teams)
   • rolled out hot fixes to all regions, completed by 15 October 2019

   View case
5. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke

   FREE NOW2019

   Symptoms

   At 16:00 CEST a developer reported in the #ask-platform Slack channel that a deployment to the Prelive cluster was failing, and a second report followed at 16:06.

   Pods PendingScale-up failure

   Trigger

   Roughly two to three days earlier, the CentOS repository mirror was restructured and container-selinux-2.68-1.el7.noarch.rpm moved, which made the static download URL hardcoded in kops nodeup invalid.

   Root cause

   Dependency failure

   Component

   Node operating system imageNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   Developers asked for help in the #ask-platform Slack channel because their deployments were failing.

   Remediation

   • downloaded the package by hand on the Prelive nodes so their bootstrap could finish
   • changed the cluster AMI from Amazon Linux 2 to Debian and applied it to all clusters

   Action items

   • Create an alert that shows when nodes cannot join the cluster, possibly from information the cluster autoscaler already has
   • Document the external dependencies of node startup and look for ways to reduce them
   • Change the AMI back to Amazon Linux 2 Minimal once kops fixes the issue
   • Document node troubleshooting better: what happens during bootstrap and where the logs are

   Lessons

   Applying cluster changes through kops and Jenkins worked well, and troubleshooting quickly showed that the failing deployments came from missing cluster capacity. What went wrong was that nobody noticed, over a long period, that some autoscaling groups held more worker instances than Kubernetes had nodes. The team was lucky that no large number of production spot instances was terminated during the problem and that it was noticed during office hours.

   View case

By root cause9