Cases7 cases

6 to 7 of 7

1. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request

   Adevinta2019

   Symptoms

   After a Spring-based JVM microservice was deployed to Kubernetes and part of production traffic was routed to it, request latency there was up to 10 times higher than on EC2. Behind the NGINX ingress controller the backend took 100 to 200 ms to answer, where the EC2 instances took about 20 ms.

   High latency

   Trigger

   Moving the service onto the Kubernetes platform, where KIAM served IAM credentials to pods with its default 15-minute lifetime.

   Root cause

   Configuration change

   Component

   Credential broker for pods

   Contributing factor

   Unreviewed defaults

   Detection

   After part of production traffic was routed to the Kubernetes deployment, its request latency was seen to be up to 10 times higher than on EC2.

   Remediation

   • reconfigured KIAM to request IAM credentials with a longer expiration period

   Lessons

   KIAM's short credential lifetime for pods and the AWS Java SDK's eager refresh of any credential within 15 minutes of expiry are each reasonable, but together they force a metadata round trip on every request. When latency rises after a move to Kubernetes, the cause is often the interaction of components that never ran together before rather than Kubernetes itself. Here a packet capture from inside the container found what latency metrics and DNS tests had not.

   View case
2. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case

By root cause1