Cases46 cases

14 to 18 of 46

1. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic

   HubSpot2024

   Symptoms

   Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

   Probe failuresHigh latencyHigh error rate

   Trigger

   Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

   Root cause

   Software defect

   Component

   Ingress controllerLiveness and readiness probes

   Contributing factor

   Automation amplificationNo headroom

   Detection

   Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

   Remediation

   • Tried to increase the capacity of the remaining load balancer instances
   • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal

   Action items

   • Change the health check configuration so health checks do not time out falsely
   • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
   • Increase the capacity of the load balancers for sudden traffic spikes
   • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
   • Update the status page automatically for large-scale outages
   • Integrate the monitoring systems more tightly with the communication tools
   • Expand load testing to simulate high-traffic scenarios
   • Introduce chaos engineering to test platform resilience regularly
   • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
   • Add auto-scaling that adjusts resource capacity to traffic patterns

   Lessons

   HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

   View case
2. IL-0148 A kube-scheduler regression in Kubernetes v1.27.14, v1.28.10 and v1.29.5 made the scheduler panic on a pod whose node affinity named a nonexistent node, so no pods were scheduled on affected AKS clusters

   Microsoft Azure2024

   Symptoms

   On AKS clusters running Kubernetes v1.27.14, v1.28.10 or v1.29.5, kube-scheduler panicked whenever the cluster contained a pod with a bad node affinity, and from then on no pod in the cluster was scheduled.

   Pods Pending

   Trigger

   A pod in the cluster with a required node affinity whose matchFields term names a node that does not exist.

   Root cause

   Software defect

   Component

   Pod scheduling and placement

   Contributing factor

   –

   Detection

   The AKS team found the bug; the issue does not say how.

   Remediation

   • AKS engineers fixed the impacted clusters
   • Rolled out a fixed kube-scheduler to all regions, applied to clusters automatically

   View case
3. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit

   Marcin Cuber2024sev1

   Symptoms

   A production EKS cluster on Kubernetes 1.30 went into read-only mode: the API server rejected every request that would change an object, such as creating pods or scaling deployments, and deleting objects to free space did not work either.

   Control plane unresponsiveControl plane lag

   Trigger

   Kyverno 1.12 kept creating ephemeralreports objects that piled up in etcd until the database passed EKS's size limit.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverPolicy engine

   Contributing factor

   No alert on limitLimited debug access

   Detection

   The cluster rejected writes; the audit logs carried the "etcdserver: mvcc: database space exceeded" error.

   Remediation

   • AWS raised the etcd storage limit from 8/10GB to 12GB to unlock writes
   • an AWS engineer deleted the leaked Kyverno ephemeralreports objects directly from etcd

   Action items

   • Monitor the etcd database size with apiserver_storage_size_bytes and never let it reach 8GB
   • Upgrade Kyverno 1.12 to 1.12.4, which fixes the ephemeralreports pile-up, or disable reporting for admission events and tune --aggregationWorkers

   Lessons

   On EKS, once etcd hits its size limit the cluster becomes read-only and only AWS can clean it up, so cleanup scripts must run before the limit is reached. Watch the etcd database size and object counts per resource, especially for controllers that create report objects.

   View case
4. IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodes

   Linode2024

   Symptoms

   At about 14:05 UTC on 22 April 2024, Linode's team noticed that new LKE clusters were failing to provision in the Toronto data center.

   Cluster or node provisioning stallIntermittent timeouts

   Trigger

   A previous update to the API did not work as intended and caused timeouts when new LKE clusters were provisioned.

   Root cause

   Software defect

   Component

   Node pool

   Contributing factor

   –

   Detection

   Linode's team noticed new LKE clusters failing to provision in the Toronto data center at about 14:05 UTC on 22 April 2024.

   Remediation

   • disabled the affected section of Toronto infrastructure from provisioning new LKE clusters
   • implemented a fix for the faulty API update
   • put the disabled infrastructure back into use after the fix

   Lessons

   The fault was in an earlier API change, not in the infrastructure that showed it, but most failures concentrated on one backend resource in one data center. Taking that resource out of provisioning limited the impact while the root cause was found, which took three days.

   View case
5. IL-0173 cert-manager's preferredChain logic picked the Let's Encrypt chain through the expired DST Root CA X3, so clients built on an old BoringSSL could not connect to a development server

   Devsisters2024

   Symptoms

   On 13 February 2024, when Devsisters' infrastructure cell came back from the Lunar New Year holiday, it received a request to look into a development server that could not be reached. No one had made a manual change over the holiday.

   TLS certificate errors

   Trigger

   On 8 February 2024, in the first phase of retiring the DST Root CA X3 cross-sign, Let's Encrypt made the ISRG Root X1 chain its default chain and began offering the DST Root CA X3 chain as the alternate.

   Root cause

   Software defect

   Component

   cert-manager

   Contributing factor

   –

   Detection

   On 13 February, after the Lunar New Year holiday, a request came in to the infrastructure cell that the development server could not be reached; while inspecting the certificate chain the server presented, the team found it serving the chain through the expired DST Root CA X3.

   Remediation

   • submitted a pull request fixing cert-manager's preferred chain selection, which was merged

   Lessons

   A latent bug can stay hidden until an upstream provider changes a default. cert-manager's preferredChain matching gave the intended result while DST Root CA X3 was Let's Encrypt's default chain and the wrong one after the default changed. An expired root at the top of a served chain is harmless to most TLS clients, which fall back to the trusted ISRG Root X1 below it, but fatal to an implementation such as an old BoringSSL that checks the trust anchor's validity and insists on verifying the topmost certificate. ACME clients choose the preferred chain with different heuristics, so the same preferredChain value can behave differently between Certbot and cert-manager.

   View case

By root cause1