Cases21 cases

1 to 5 of 21

1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable

   ckck8032026

   Symptoms

   The cluster could not be reached from outside. On the control plane node, sudo systemctl status kubelet showed TLS handshake errors such as "http: TLS handshake error from 101.198.0.183:19517: tls: client requested unsupported application protocols" and "http: TLS handshake error from 101.198.0.172:36328: tls: client offered only unsupported versions: [302 301]".

   Control plane unresponsiveContainer crash loop

   Trigger

   The leaf certificates that kubeadm had generated for the cluster, valid for one year by default, expired at 2026-02-20 16:16 UTC.

   Root cause

   Certificate or credential expiry

   Component

   Kubernetes API serveretcdkubeletCalico

   Contributing factor

   Unreviewed defaults

   Detection

   The author could not connect to the cluster from outside and went to the control plane node to read the kubelet logs.

   Remediation

   • renewed all expired certificates with sudo kubeadm certs renew all
   • restarted containerd and the kubelet so the static control plane pods restarted with the new certificates
   • copied the renewed /etc/kubernetes/admin.conf to $HOME/.kube/config

   Lessons

   A kubeadm cluster stops working one year after its leaf certificates were issued unless they are renewed. The first errors seen may be TLS handshake failures or a flag error that does not mention certificates; the decisive message is the x509 expiry in the kube-apiserver log for its connection to etcd. The author recommends checking expiry with kubeadm certs check-expiration and renewing from 30 days before. kubeadm upgrade apply renews the certificates too, so a cluster upgraded at least once a year avoids the problem; a cluster run for a long time without upgrades can renew from cron, for example with 0 3 1 1 * root kubeadm certs renew all && systemctl restart kubelet in /etc/cron.d/kubeadm-cert-renew.

   View case
2. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries

   PostHog2025

   Symptoms

   On 21 October 2025 at 21:45 UTC a change to timeout handling was deployed to the feature flags service, and two minutes later automated monitoring reported higher error rates. The change was rolled back by 21:49, but error rates stayed high: about 38% of flag evaluation requests in the US datacenter returned errors. At 22:30 Redis metrics showed memory exhaustion on ElastiCache, at 22:35 Postgres connections spiked, and at 22:45 the team found a massive data transfer from Postgres into Redis. The main PostHog application was affected as well, because it shares the Redis instance with the flags service.

   Sustained high node CPUHigh error rateHigh latencyContainer crash loopDatabase overload

   Trigger

   On 21 October a deployment meant to reduce timeout errors (PR #39821); on 24 October the deployment of IP-based rate limiting (PR #40074) that saw all traffic as coming from the load balancer's IP; on 28 October an unrelated routine deployment that rolled the feature flag pods in the US region.

   Root cause

   Capacity exhaustion

   Component

   Node poolPod scheduling and placementIn-memory cacheDatabase

   Contributing factor

   No alert on limitRetry stormIneffective rollbackSingle point of failureNo staged rolloutUnrepresentative staging

   Detection

   On 21 October automated monitoring detected higher error rates two minutes after the deploy. On 24 October no alert existed for 429 errors, and the team learned of the incident from customer reports that began at 18:10 UTC, a 62-minute detection delay. On 28 October on-call received a high error count alert. On 29 October the incident was reported with higher error rates and latency. No alert existed for CPU pressure, so the CPU saturation behind all three incidents was not seen directly.

   Remediation

   • rolled back the timeout handling change within 2 minutes on 21 October, which did not stop the errors
   • throttled connections and Redis writes on 21 October
   • halted the rollout on 28 October, scaled the US flags pods to zero until Redis recovered, then scaled them back gradually
   • rolled back hardware changes on 30 October at 00:03 UTC, which removed most errors but not the latency
   • increased the pod resource requests and adjusted the Kubernetes fleet size so fewer pods ran on each node
   • added retry limits to Envoy between the load balancer and the service
   • moved database connection timeouts and other critical settings to runtime configuration and raised the timeout values
   • added per-node and per-pod CPU alerts at 80% sustained for 5 minutes that page on-call

   Action items

   • Form a strike team from the flags, ingestion and infrastructure teams to review the application and infrastructure for remaining bottlenecks
   • Investigate separating the flags Redis instance from the application Redis instance
   • Before re-enabling ArgoCD sync, align the flags service chart configuration with the values in ArgoCD and define the deployment and Redis strategy
   • Make the feature flags team hard code-owners for flags-related code
   • Build a dashboard of key flag metrics with runbook links and update the feature flag runbooks
   • Add missing alerts on existing service and infrastructure metrics
   • Make readiness checks validate dependencies that degrade under load, for example by pinging the database
   • Rate-limit cache population so it cannot overwhelm Redis
   • Monitor connection pools with automatic throttling near exhaustion and limit concurrent connections
   • Build a production-scale load testing framework
   • Build progressive rollout infrastructure to limit blast radius
   • Document Postgres-to-Redis data flows and write runbooks for data transfer storms

   Lessons

   Nodes that are small relative to pod requests let the scheduler pack many pods on each node, and the resulting CPU saturation showed up three different ways: excessive parallelism, connection pool exhaustion and slow queries. Treating each symptom separately let the problem return; right-sizing the pods and nodes fixed all three. CPU must be alerted on from the start. Unbounded Envoy retries and unbounded cache population amplified the overload, and a Redis instance shared with the main application spread a flags problem to the main application. A rollback did not end the incident, because the data transfer and resource exhaustion it had started kept running.

   View case
3. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop

   Yonghyun Kim2025

   Symptoms

   While the Spring Cloud Config Server was briefly down for an upgrade, every client application began to answer its liveness probe with a failure saying it could not reach the config server.

   Probe failuresContainer crash loopService outage

   Trigger

   A downtime upgrade of the Spring Cloud Config Server made it stop responding for a short time.

   Root cause

   Dependency failure

   Component

   Liveness and readiness probes

   Contributing factor

   Unreviewed defaultsAutomation amplificationExcessive blast radius

   Lessons

   A liveness probe that includes an external dependency turns a short outage of that dependency into a restart loop of every client. Keep external dependencies out of liveness checks and handle temporary problems with readiness probes. Libraries can add health indicators to Spring Boot's actuator health endpoint automatically, so check what the probe path actually reports. The author ran another Kubernetes environment for two years without liveness probes, relying on strict readiness probes and on the application terminating itself on unrecoverable faults.

   View case
4. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours

   Wealthy2025sev1

   Symptoms

   Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

   Scale-up failureContainer crash loopService outageDNS resolution failures

   Trigger

   Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

   Root cause

   Breaking change in an upgrade

   Component

   GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets

   Contributing factor

   Deferred upgradesSingle point of failureIneffective rollbackNo headroom

   Remediation

   • Created a new GKE cluster at v1.33, the version of the forced upgrade
   • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
   • Updated the cert-manager configuration for Kubernetes 1.33
   • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
   • Decommissioned the broken cluster

   Action items

   • Configure all clusters to use the GKE stable release channel
   • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
   • Review spot instance usage for critical workloads and add on-demand fallback pools
   • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
   • Document a Kubernetes, Istio and cert-manager compatibility matrix
   • Create an automated upgrade testing pipeline
   • Reserve static IPs for quick migrations
   • Pin versions of Istio, cert-manager, CSI and DNS addons
   • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines

   Lessons

   GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

   View case
5. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates

   OpenAI2025

   Symptoms

   From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

   Container crash loopHigh error rate

   Trigger

   An internal engineer applied a configuration change that contained an invalid value.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   Excessive blast radiusNo staged rollout

   Remediation

   • reverted the configuration to the correct value
   • started a rolling restart of the affected pods to force them to resynchronise

   Action items

   • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
   • Move to a service-specific configuration value instead of one shared between services
   • Improve deployment safeguards

   View case

By root cause8