Cases46 cases

19 to 23 of 46

1. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together

   UK Ministry of Justice2023

   Symptoms

   At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.

   Service outage

   Trigger

   The team changed the instance type of the cluster's worker nodes through Terraform.

   Root cause

   Configuration change

   Component

   Node poolTerraform

   Contributing factor

   Excessive blast radiusNo runbook

   Detection

   Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.

   Action items

   • Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
   • Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type

   View case
2. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet

   Aqua Security2023

   Symptoms

   Aqua's research team Nautilus recorded the attack on one of its Kubernetes honeypots. What appeared in the cluster was a DaemonSet running the image kuberntesio/kube-controller:1.0.1 from Docker Hub on every node, using the nodes' CPU to mine Monero.

   Unauthorised access or a foreign workloadResources deleted

   Trigger

   An API server misconfigured to accept unauthenticated requests from anonymous users with privileges.

   Root cause

   Configuration change

   Component

   Kubernetes API serverRBAC and service accountsContainer registry and its credentials

   Contributing factor

   Monitoring blind spot

   Detection

   Aqua recorded the attacker's API requests on its honeypot; planted AWS access keys raised a beacon when the attacker tried to use them.

   Lessons

   Fixing an exposed API server is not enough once an attacker has been in: a ClusterRoleBinding to an attacker-created ServiceAccount keeps working after anonymous access is closed. Attackers name these objects and images after real Kubernetes components so they blend into the API audit logs and pass for legitimate deployments. Aqua recommends scanning clusters for vulnerabilities, exposed secrets and misconfigurations with a tool such as Trivy.

   View case
3. IL-0186 Pods on one Kubernetes worker node could not reach external HTTP and HTTPS because storage components had added nat PREROUTING rules that redirected ports 80 and 443 to local ports

   Hans H. Kim2023

   Symptoms

   On one worker node of a Kubernetes cluster, traffic to ports 80 and 443 went missing: pods on that node could not make outbound HTTP or HTTPS requests. Infrastructure engineers and the author's teammates debugged it.

   Misrouted requests

   Trigger

   A server on which components for storage work had been installed automatically added nat REDIRECT rules for ports 80 and 443 to the node's iptables.

   Root cause

   Configuration change

   Component

   Node iptables rules

   Contributing factor

   –

   Remediation

   • had the people responsible for the storage components delete the REDIRECT rules from the node

   Lessons

   Inside Kubernetes, kube-proxy itself manages iptables rules, so the author felt that part needs to be understood in depth. When packets go missing during network tests, check each step of the path in turn, which requires knowing clearly which steps the networking goes through.

   View case
4. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down

   Stytch2023

   Symptoms

   On 23 February 2023 Stytch's EKS worker nodes went Ready:Unknown one after another, Karpenter failed to launch replacement nodes, and with fewer and fewer pods scheduled the Live Stytch API went down for 14 minutes, returning 5XX on every call.

   Node NotReadyScale-up failurePod evictionPods PendingService outage

   Trigger

   At 14:21 UTC Stytch landed a change to remove unused EKS managed node groups, because it had decided to run its data plane applications fully on nodes provisioned by Karpenter. Because Crossplane resources use an orphan deletion policy, engineers deleted the node groups in the AWS console, first in staging and then in production.

   Root cause

   Configuration change

   Component

   KarpenterNode poolConfiguration templating

   Contributing factor

   Unverified assumptionShared credentialChange with a delayed effectIgnored early signal

   Detection

   Low-urgency alerts that Karpenter could not find the instance profile fired in staging at 16:06 UTC and in production at 17:48 UTC but were not treated as an incident. The incident process started at 20:26 UTC when node launch failures spiked the Karpenter error rate, and Kubernetes monitoring alerts in production followed at 20:44 UTC. The support team flagged the outage at 21:14 UTC.

   Remediation

   • Reverted the node group deletion, which recreated the managed node groups
   • Manually scaled up one of the Live API node groups with more and much larger nodes and removed its taints
   • Removed node selectors from the applications one after another so their pods could be scheduled

   Action items

   • Treat Karpenter errors as high-urgency alerts that page the on-call engineer
   • Tune cluster alerting thresholds to send a high-urgency page earlier
   • Add alerts for nodes that stay NotReady or Unknown for long periods
   • Raise the thresholds of other deployment alerts, such as unschedulable pods and replicas not available
   • Give Karpenter its own IAM instance profile, separate from the one used by the node groups
   • Separate other AWS resources that were shared between uses, specifically IAM and security groups
   • Rework the EKS and Karpenter configuration so system instance profiles are unique, static per cluster and generated by Karpenter
   • Introduce a cloud visualization tool for engineers making infrastructure changes
   • Add live system and product metrics to the status page
   • Ask AWS to confirm the undocumented actions and side effects of deleting a node group
   • Correct the shared use of IAM roles and security groups across the infrastructure

   Lessons

   The IAM role for Karpenter's self-managed nodes should never have been the same as the one for the managed node groups; sharing it is what let the node group deletion break Karpenter, and the roles had been left combined for speed and simplicity when Karpenter was first set up. The hours-long delay between the configuration change and its effect made the incident hard to assess, and AWS confirmed that this delay is expected, so Stytch will allow for it in future IAM instance profile changes.

   View case
5. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes

   DFDS2023sev1

   Symptoms

   At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.

   Service outageResources deletedMonitoring outage

   Trigger

   A scheduled node rollover had accidentally made containerd the default container runtime, which stopped log shipping to CloudWatch; the unscheduled rollover to revert it timed out, and the pipeline run that followed reduced the Auto Scaling group from 30 to 15 instances.

   Root cause

   Configuration change

   Component

   Container runtimeNode launch templateNode poolTerraformTelemetry agent

   Contributing factor

   Automation amplificationUnreviewed defaultsUnrepresentative stagingNo configuration testsNo runbookHarmful manual action

   Detection

   End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.

   Remediation

   • ran the node rollover script to add new nodes, which failed on a timeout from Docker Hub
   • ran the node rollover script a second time, which added new nodes that the load balancer marked healthy at 11:01 UTC

   Action items

   • Document the rollover process, especially what can be done when it goes wrong
   • Investigate options for improving the rollover process
   • Add fluentd log collection to the QA test suite so logs are covered end to end
   • Add alarms for fluentd when too little or no log data is sent
   • Implement a staging environment
   • Implement guards against selecting an invalid container runtime

   Lessons

   When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way. The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.

   View case

By root cause1