Incident Library
Applied filters SymptomDNS resolution failuresFactorUnpatched known bugOrganisationRecruitClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0580 Deploys on a ConcourseCI pipeline failed intermittently with DNS lookup timeouts when pulling images from Amazon ECR
    Recruit2019
    Symptoms

    When an engineer at Recruit Technologies pressed the deploy button, Docker running in a ConcourseCI job failed to pull an image from Amazon ECR that docker build uses as cache. The error was: Error response from daemon: Get https://xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/v2/: dial tcp: lookup xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com on 172.18.0.3:53: read udp 10.254.0.122:48372->172.18.0.3:53: i/o timeout

    DNS resolution failuresImage pull failures
    Trigger

    The Elastic Load Balancer behind the ECR endpoint returned more A records, as an ELB does when its units grow with request volume, which made the DNS responses larger.

    Root cause
    Software defect
    Component
    CI systemContainer registry and its credentialsClient-side DNS resolver and cache
    Contributing factor
    Unpatched known bug
    Detection

    The deploy failed with the DNS lookup timeout in the CI job when an engineer pressed the deploy button.

    Remediation
    • Changed the nameserver in resolv.conf on the CI worker to another DNS cache server as an interim fix
    Action items
    • Filed an issue and a pull request to Concourse to fix its DNS proxy forwarding uncompressed responses
    View case
Check your cluster for this

This case involves CI system and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
CI systemClient-side DNS resolver and cacheContainer registry and its credentials
Contributing factor
Unpatched known bug
Organisation
Recruit
  1. IL-0580 Deploys on a ConcourseCI pipeline failed intermittently with DNS lookup timeouts when pulling images from Amazon ECR
    Recruit2019