Incident Library
Applied filters ComponentCredential broker for podsFactorUnreviewed defaultsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019
    Symptoms

    After a Spring-based JVM microservice was deployed to Kubernetes and part of production traffic was routed to it, request latency there was up to 10 times higher than on EC2. Behind the NGINX ingress controller the backend took 100 to 200 ms to answer, where the EC2 instances took about 20 ms.

    High latency
    Trigger

    Moving the service onto the Kubernetes platform, where KIAM served IAM credentials to pods with its default 15-minute lifetime.

    Root cause
    Configuration change
    Component
    Credential broker for pods
    Contributing factor
    Unreviewed defaults
    Detection

    After part of production traffic was routed to the Kubernetes deployment, its request latency was seen to be up to 10 times higher than on EC2.

    Remediation
    • reconfigured KIAM to request IAM credentials with a longer expiration period
    Lessons

    KIAM's short credential lifetime for pods and the AWS Java SDK's eager refresh of any credential within 15 minutes of expiry are each reasonable, but together they force a metadata round trip on every request. When latency rises after a move to Kubernetes, the cause is often the interaction of components that never ran together before rather than Kubernetes itself. Here a packet capture from inside the container found what latency metrics and DNS tests had not.

    View case
  2. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
    Symptoms

    Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

    High error rateCredentials unavailableControl plane lag
    Trigger

    An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

    Root cause
    Capacity exhaustion
    Component
    kubeletCredential broker for podsKubernetes API serverCI system
    Contributing factor
    Unreviewed defaultsSingle point of failure
    Remediation
    • Scaled the CDP cluster up by hand so builder pods spread over more than one node
    Action items
    • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration
    View case
Check your cluster for this

These 2 cases involve Credential broker for pods and CI system. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Configuration change1 case

Component
Credential broker for pods
Contributing factor
Unreviewed defaults
Organisation
Adevinta
  1. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019

Capacity exhaustion1 case

Component
CI systemCredential broker for podsKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsSingle point of failure
Organisation
Zalando
  1. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019