Cases1 case

1 to 1 of 1

1. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down

   Stytch2023

   Symptoms

   On 23 February 2023 Stytch's EKS worker nodes went Ready:Unknown one after another, Karpenter failed to launch replacement nodes, and with fewer and fewer pods scheduled the Live Stytch API went down for 14 minutes, returning 5XX on every call.

   Node NotReadyScale-up failurePod evictionPods PendingService outage

   Trigger

   At 14:21 UTC Stytch landed a change to remove unused EKS managed node groups, because it had decided to run its data plane applications fully on nodes provisioned by Karpenter. Because Crossplane resources use an orphan deletion policy, engineers deleted the node groups in the AWS console, first in staging and then in production.

   Root cause

   Configuration change

   Component

   KarpenterNode poolConfiguration templating

   Contributing factor

   Unverified assumptionShared credentialChange with a delayed effectIgnored early signal

   Detection

   Low-urgency alerts that Karpenter could not find the instance profile fired in staging at 16:06 UTC and in production at 17:48 UTC but were not treated as an incident. The incident process started at 20:26 UTC when node launch failures spiked the Karpenter error rate, and Kubernetes monitoring alerts in production followed at 20:44 UTC. The support team flagged the outage at 21:14 UTC.

   Remediation

   • Reverted the node group deletion, which recreated the managed node groups
   • Manually scaled up one of the Live API node groups with more and much larger nodes and removed its taints
   • Removed node selectors from the applications one after another so their pods could be scheduled

   Action items

   • Treat Karpenter errors as high-urgency alerts that page the on-call engineer
   • Tune cluster alerting thresholds to send a high-urgency page earlier
   • Add alerts for nodes that stay NotReady or Unknown for long periods
   • Raise the thresholds of other deployment alerts, such as unschedulable pods and replicas not available
   • Give Karpenter its own IAM instance profile, separate from the one used by the node groups
   • Separate other AWS resources that were shared between uses, specifically IAM and security groups
   • Rework the EKS and Karpenter configuration so system instance profiles are unique, static per cluster and generated by Karpenter
   • Introduce a cloud visualization tool for engineers making infrastructure changes
   • Add live system and product metrics to the status page
   • Ask AWS to confirm the undocumented actions and side effects of deleting a node group
   • Correct the shared use of IAM roles and security groups across the infrastructure

   Lessons

   The IAM role for Karpenter's self-managed nodes should never have been the same as the one for the managed node groups; sharing it is what let the node group deletion break Karpenter, and the roles had been left combined for speed and simplicity when Karpenter was first set up. The hours-long delay between the configuration change and its effect made the incident hard to assess, and AWS confirmed that this delay is expected, so Stytch will allow for it in future IAM instance profile changes.

   View case

By root cause1