Incident Library

IL-0231 Stytch · 2023-02-23kubernetesaws

Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down

Stytch runs on AWS EKS and creates its clusters, managed node groups and other cloud resources with Crossplane from a management cluster. After moving from Fargate to EC2 nodes, first on statically sized managed node groups and then on nodes provisioned by Karpenter, Stytch decided to remove the managed node groups for its data plane. Two engineers deleted the node groups in the AWS console, staging first. The console warned that deleting a managed node group removes the worker role from the aws-auth ConfigMap; the engineers expected Crossplane, which manages that ConfigMap, to put the role back at once, saw no impact in staging and repeated the change in production. In the background the AWS API ran cleanup calls after the node group deletion, and CloudTrail later showed that the instance profile was deleted, at 14:24 UTC in staging and 14:50 UTC in production. Karpenter's nodes used the same IAM role and instance profile as the node groups.

Crossplane recreated the role, but its Cluster Composition gave the new instance profile a randomly generated name instead of the old name that Karpenter's configuration referred to, so Karpenter had no permission to launch nodes. Nothing broke until the deleted instance profile was needed again hours later. From 16:06 UTC low-urgency warnings about the missing instance profile came from staging and then production; staging nodes began shutting down at 19:54 UTC, and from 20:13 UTC production nodes went Ready:Unknown, were tainted, had their pods evicted and were not replaced. Stytch started its incident process and froze deployments at 20:26 UTC.

At 21:00 UTC the team identified the node group change and reverted the deletion, which recreated the node groups, but the new nodes came up too slowly and the API went down at 21:10 UTC. The team then manually scaled up one of the Live API node groups with more and much larger nodes, removed its taints, and removed node selectors from the applications one after another so they could be scheduled. The Live API returned at 21:24 UTC, the Dashboard and Frontend SDKs at 21:34 UTC and the Test API at 21:36 UTC. AWS later confirmed the undocumented cascade of the DeleteNodegroup API: worker nodes are drained, pods on them are evicted, the service role deletes resources associated with the node group such as roles, policies and security groups, and the worker instances are terminated.

Severity
–
Scope
Full outage of the Live Stytch API for 14 minutes and of the Stytch Frontend SDKs and Dashboard for 17 minutes; the production Test API was down from 21:10 to 21:36 UTC
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On 23 February 2023 Stytch's EKS worker nodes went Ready:Unknown one after another, Karpenter failed to launch replacement nodes, and with fewer and fewer pods scheduled the Live Stytch API went down for 14 minutes, returning 5XX on every call.

The first signs came hours earlier as low-urgency warnings that Karpenter could not find the specified instance profile and could not launch a new node: at 16:06 UTC in staging and at 17:48 UTC in production. The on-call engineer judged both low urgency because existing deployments were healthy and new pods were still being scheduled. At 19:54 UTC existing staging nodes began shutting down and kube-proxy health notifications arrived from staging, while production still looked healthy.

At 20:13 UTC the first production node was marked Ready:Unknown by the Kubernetes control plane. Nodes running with the deleted instance profile terminated slowly as they tried to use it; after five minutes in Unknown the taint controller added an unreachable taint and all pods on the node were evicted. Karpenter kept trying to launch new nodes with the deleted instance profile, and at 20:26 UTC its error rate spiked. Kubernetes monitoring alerts fired in production at 20:44 UTC and Deployment Replicas Unavailable alerts at 20:48 UTC; the Live and Test API went down at 21:10 UTC and the Dashboard and Frontend SDKs at 21:17 UTC.

Reverting the node group deletion at about 21:00 UTC recreated the managed node groups, but new pods could not be scheduled quickly enough because they had to wait for the new nodes to start.

Symptom Node NotReadyScale-up failurePod evictionPods PendingService outage

Root cause

Root cause Configuration change

Component KarpenterNode poolConfiguration templating
Contributing factor Unverified assumptionShared credentialChange with a delayed effectIgnored early signal
Trigger
At 14:21 UTC Stytch landed a change to remove unused EKS managed node groups, because it had decided to run its data plane applications fully on nodes provisioned by Karpenter. Because Crossplane resources use an orphan deletion policy, engineers deleted the node groups in the AWS console, first in staging and then in production.

Detection

Low-urgency alerts that Karpenter could not find the instance profile fired in staging at 16:06 UTC and in production at 17:48 UTC but were not treated as an incident. The incident process started at 20:26 UTC when node launch failures spiked the Karpenter error rate, and Kubernetes monitoring alerts in production followed at 20:44 UTC. The support team flagged the outage at 21:14 UTC.

Remediation

Action items

Lessons

The IAM role for Karpenter's self-managed nodes should never have been the same as the one for the managed node groups; sharing it is what let the node group deletion break Karpenter, and the roles had been left combined for speed and simplicity when Karpenter was first set up. The hours-long delay between the configuration change and its effect made the incident hard to assess, and AWS confirmed that this delay is expected, so Stytch will allow for it in future IAM instance profile changes.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Karpenter, Node pool, Configuration templating among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.