Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
    Symptoms

    Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

    High error rateCredentials unavailableControl plane lag
    Trigger

    An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

    Root cause
    Capacity exhaustion
    Component
    kubeletCredential broker for podsKubernetes API serverCI system
    Contributing factor
    Unreviewed defaultsSingle point of failure
    Remediation
    • Scaled the CDP cluster up by hand so builder pods spread over more than one node
    Action items
    • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration
    View case
Check your cluster for this

This case involves CI system and Credential broker for pods. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
CI systemCredential broker for podsKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsSingle point of failure
Organisation
Zalando
  1. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019