Cases1 case

1 to 1 of 1

1. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request

   Adevinta2019

   Symptoms

   After a Spring-based JVM microservice was deployed to Kubernetes and part of production traffic was routed to it, request latency there was up to 10 times higher than on EC2. Behind the NGINX ingress controller the backend took 100 to 200 ms to answer, where the EC2 instances took about 20 ms.

   High latency

   Trigger

   Moving the service onto the Kubernetes platform, where KIAM served IAM credentials to pods with its default 15-minute lifetime.

   Root cause

   Configuration change

   Component

   Credential broker for pods

   Contributing factor

   Unreviewed defaults

   Detection

   After part of production traffic was routed to the Kubernetes deployment, its request latency was seen to be up to 10 times higher than on EC2.

   Remediation

   • reconfigured KIAM to request IAM credentials with a longer expiration period

   Lessons

   KIAM's short credential lifetime for pods and the AWS Java SDK's eager refresh of any credential within 15 minutes of expiry are each reasonable, but together they force a metadata round trip on every request. When latency rises after a move to Kubernetes, the cause is often the interaction of components that never ran together before rather than Kubernetes itself. Here a packet capture from inside the container found what latency metrics and DNS tests had not.

   View case

By root cause1