Cases2 cases

1 to 2 of 2

1. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped

   Google Cloud2024

   Symptoms

   From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

   Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate

   Trigger

   Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

   Root cause

   Software defect

   Component

   Network control plane (SDN)Cloud load balancerNAT gateway

   Contributing factor

   Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius

   Detection

   Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

   Remediation

   • Paused maintenance operations to prevent the shutdown from repeating
   • Brought the affected VPC Controllers back into a serving state in every zone
   • Stopped the automation responsible for the shutdown until safeguards are in place
   • Recovered partial Apigee proxy deployments by hand
   • Cleaned up VMs left behind by failed Dataproc cluster deletions

   Action items

   • Add safety measures that check that maintenance operations carry correctly scoped zone information
   • Make the underlying tools reject maintenance operations that do not specify zones
   • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
   • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures

   View case
2. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down

   Stytch2023

   Symptoms

   On 23 February 2023 Stytch's EKS worker nodes went Ready:Unknown one after another, Karpenter failed to launch replacement nodes, and with fewer and fewer pods scheduled the Live Stytch API went down for 14 minutes, returning 5XX on every call.

   Node NotReadyScale-up failurePod evictionPods PendingService outage

   Trigger

   At 14:21 UTC Stytch landed a change to remove unused EKS managed node groups, because it had decided to run its data plane applications fully on nodes provisioned by Karpenter. Because Crossplane resources use an orphan deletion policy, engineers deleted the node groups in the AWS console, first in staging and then in production.

   Root cause

   Configuration change

   Component

   KarpenterNode poolConfiguration templating

   Contributing factor

   Unverified assumptionShared credentialChange with a delayed effectIgnored early signal

   Detection

   Low-urgency alerts that Karpenter could not find the instance profile fired in staging at 16:06 UTC and in production at 17:48 UTC but were not treated as an incident. The incident process started at 20:26 UTC when node launch failures spiked the Karpenter error rate, and Kubernetes monitoring alerts in production followed at 20:44 UTC. The support team flagged the outage at 21:14 UTC.

   Remediation

   • Reverted the node group deletion, which recreated the managed node groups
   • Manually scaled up one of the Live API node groups with more and much larger nodes and removed its taints
   • Removed node selectors from the applications one after another so their pods could be scheduled

   Action items

   • Treat Karpenter errors as high-urgency alerts that page the on-call engineer
   • Tune cluster alerting thresholds to send a high-urgency page earlier
   • Add alerts for nodes that stay NotReady or Unknown for long periods
   • Raise the thresholds of other deployment alerts, such as unschedulable pods and replicas not available
   • Give Karpenter its own IAM instance profile, separate from the one used by the node groups
   • Separate other AWS resources that were shared between uses, specifically IAM and security groups
   • Rework the EKS and Karpenter configuration so system instance profiles are unique, static per cluster and generated by Karpenter
   • Introduce a cloud visualization tool for engineers making infrastructure changes
   • Add live system and product metrics to the status page
   • Ask AWS to confirm the undocumented actions and side effects of deleting a node group
   • Correct the shared use of IAM roles and security groups across the infrastructure

   Lessons

   The IAM role for Karpenter's self-managed nodes should never have been the same as the one for the managed node groups; sharing it is what let the node group deletion break Karpenter, and the roles had been left combined for speed and simplicity when Karpenter was first set up. The hours-long delay between the configuration change and its effect made the incident hard to assess, and AWS confirmed that this delay is expected, so Stytch will allow for it in future IAM instance profile changes.

   View case

By root cause2