Cases45 cases

16 to 20 of 45

1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU

   Namespace2024

   Symptoms

   From 17 October 2024, packets through the vSwitch that joined Namespace's bare metal machines at one provider in the fra3 region were dropped at random. Retries in TCP, QUIC and the Namespace control plane hid most of it, but the SLO error budget for latency targets on key interactions began to burn.

   Intermittent connection failuresHigh latencyControl plane lagService outage

   Trigger

   Random packet drops on the bare metal provider's vSwitch from 17 October 2024; for the second outage, a rate of Kubernetes changes that the slow NVMe on one control plane node could no longer keep up with.

   Root cause

   Cloud provider outage

   Component

   Kubernetes API server

   Contributing factor

   No capacity planningNo headroomOverlapping failures

   Detection

   SLO-based monitoring of latency targets showed the error budget burning while retries hid the individual packet drops.

   Remediation

   • diverted all customer traffic from fra3 to other regions
   • connected each provider machine to the zrh1 cluster over Wireguard instead of the vSwitch
   • repaved every fra3 machine with a new identity, certificates and encryption-at-rest keys, dropping regional caches to recover faster
   • cloned existing datasets in a more distributed way to speed up repaving the machines
   • added machines back to the new region in batches and returned the region to the compute pool
   • redirected workloads away from zrh1 during the second outage
   • cloned the failing control plane node's key state to a higher-capacity node that took over its role and IP address

   Action items

   • Credit all instances that ran during the outages back to customers
   • Build up backup regions and solutions so a similar event has options that do not affect customers

   Lessons

   Running hot with a delayed hardware delivery left no room to take a region out, so fixing a provider network fault turned into a global capacity shortage. After one outage the team was anchored on its cause and first looked for network problems in the second outage, which had a different cause: a control plane node whose disk could not keep up with the Kubernetes change rate.

   View case
2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one

   Wikimedia Foundation2024

   Symptoms

   At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

   High error rateHigh latencyContainer crash loopProbe failuresControl plane lag

   Trigger

   An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

   Root cause

   Capacity exhaustion

   Component

   CalicoHelmJobs and CronJobs

   Contributing factor

   Change with a delayed effectNo runbook

   Detection

   Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

   Remediation

   • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods

   Action items

   • Stop mwscript-k8s from creating so many resources

   Lessons

   The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

   View case
3. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic

   HubSpot2024

   Symptoms

   Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

   Probe failuresHigh latencyHigh error rate

   Trigger

   Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

   Root cause

   Software defect

   Component

   Ingress controllerLiveness and readiness probes

   Contributing factor

   Automation amplificationNo headroom

   Detection

   Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

   Remediation

   • Tried to increase the capacity of the remaining load balancer instances
   • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal

   Action items

   • Change the health check configuration so health checks do not time out falsely
   • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
   • Increase the capacity of the load balancers for sudden traffic spikes
   • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
   • Update the status page automatically for large-scale outages
   • Integrate the monitoring systems more tightly with the communication tools
   • Expand load testing to simulate high-traffic scenarios
   • Introduce chaos engineering to test platform resilience regularly
   • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
   • Add auto-scaling that adjusts resource capacity to traffic patterns

   Lessons

   HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

   View case
4. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation

   이언철2024

   Symptoms

   When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.

   High latencyIntermittent connection failuresAutoscaler misbehaviour

   Trigger

   Sharp changes in traffic loaded the small number of CoreDNS pods beyond what they could handle.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSKarpenterHorizontal pod autoscalerPod scheduling and placement

   Contributing factor

   Automation amplificationHarmful manual action

   Remediation

   • Set up the Cluster Proportional Autoscaler for CoreDNS with nodesPerReplica and coresPerReplica chosen for the instance types in use
   • Spread CoreDNS with soft topologySpreadConstraints (ScheduleAnyway) instead of anti-affinity so Karpenter could consolidate nodes
   • Added alert thresholds on CoreDNS error rate, panics, request duration and request rate
   • Deployed NodeLocal DNSCache so each node answers DNS from a local cache and forwards misses to the main CoreDNS

   Lessons

   A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.

   View case
5. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover

   Honeycomb2023sev1

   Symptoms

   Standard alerting paged Honeycomb's on-call engineers about the performance and stability of event ingestion, and the automated load-shedding mechanisms tripped and sent further notifications. The status page described a higher than normal error rate and response time on the ingestion service.

   High error rateHigh latencyContainer crash loopService outageDatabase overload

   Trigger

   An abnormally large number of queries arrived from many separate connections within a few minutes.

   Root cause

   Capacity exhaustion

   Component

   DatabaseIn-memory cache

   Contributing factor

   Retry stormAutomation amplification

   Detection

   Standard alerting notified the on-call engineers about ingestion performance and stability, and the automated load-shedding mechanisms added notifications when they tripped. The first status post, about alerts on a higher than normal error rate and response time, went up at 18:28 PST (02:28 UTC), 13 minutes after the 02:15 UTC start.

   Remediation

   • Scaled the ingest fleet up manually and aggressively
   • Cut off most ingest for a few minutes so the ingest hosts could come back with enough capacity
   • Temporarily lowered the rate limits for the most aggressive traffic sources until their traffic subsided

   Action items

   • Keep the ingest fleet scaled up to prevent a repeat while longer-term fixes are worked on
   • Decide on the projects that will address these failure paths

   Lessons

   Load shedding and auto-scaling were in place, yet a burst that arrived faster than the cache could spread its writes still took the ingest hosts down. During recovery, Kubernetes' crash loop back-off worked against the operators: the hosts that had failed stayed offline while the fleet needed every host. Clients that retry whole OTel batches after a partial drop can multiply traffic many times over, so per-source rate limits were what finally stabilised ingest.

   View case

By root cause7