Incident Library
Applied filters ComponentClient-side DNS resolver and cacheFactorUnpatched known bugClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0580 Deploys on a ConcourseCI pipeline failed intermittently with DNS lookup timeouts when pulling images from Amazon ECR
    Recruit2019
    Symptoms

    When an engineer at Recruit Technologies pressed the deploy button, Docker running in a ConcourseCI job failed to pull an image from Amazon ECR that docker build uses as cache. The error was: Error response from daemon: Get https://xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/v2/: dial tcp: lookup xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com on 172.18.0.3:53: read udp 10.254.0.122:48372->172.18.0.3:53: i/o timeout

    DNS resolution failuresImage pull failures
    Trigger

    The Elastic Load Balancer behind the ECR endpoint returned more A records, as an ELB does when its units grow with request volume, which made the DNS responses larger.

    Root cause
    Software defect
    Component
    CI systemContainer registry and its credentialsClient-side DNS resolver and cache
    Contributing factor
    Unpatched known bug
    Detection

    The deploy failed with the DNS lookup timeout in the CI job when an engineer pressed the deploy button.

    Remediation
    • Changed the nameserver in resolv.conf on the CI worker to another DNS cache server as an interim fix
    Action items
    • Filed an issue and a pull request to Concourse to fix its DNS proxy forwarding uncompressed responses
    View case
  2. IL-0454 Baemin billing servers could not open new MariaDB connections at the start of a card-discount event
    Woowa Brothers2016
    Symptoms

    At 18:00, when a card-company event giving 3,000 won off to the first 1,000 customers started, the Pinpoint graph of the Baemin billing system turned red shortly afterwards: 140 of 14,000 requests failed in about one minute. Most errors looked like requests from the billing system that the payment gateway (PG) did not receive because of a network problem, but other errors pointed at the billing system itself.

    High error rateIntermittent connection failures
    Trigger

    A card-company discount event for the first 1,000 customers started at 18:00 and sent about 14,000 requests to the billing system in about one minute.

    Root cause
    Software defect
    Component
    DatabaseClient-side DNS resolver and cache
    Contributing factor
    Unpatched known bug
    Detection

    The team was watching NewRelic and Pinpoint during the 18:00 event, and the Pinpoint graph turned red shortly after the event began.

    Remediation
    • Changed the Gradle settings to use mariadb-connector-j 1.5.5 instead of 1.4.6
    Lessons

    A new DB connection failed because the driver did a DNS lookup of the local host name that took about 6 seconds, longer than the server's 5-second connect_timeout. Other ways to avoid it are to use localhost as the host name or to register the IP address and host name in /etc/hosts so the address is returned first. Above all, the author recommends keeping libraries on recent versions.

    View case
Check your cluster for this

These 2 cases involve Client-side DNS resolver and cache and CI system. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Client-side DNS resolver and cache2CI system1Database1Container registry and its credentials1
Contributing factor
Unpatched known bug2
Organisation
Recruit1Woowa Brothers1
  1. IL-0580 Deploys on a ConcourseCI pipeline failed intermittently with DNS lookup timeouts when pulling images from Amazon ECR
    Recruit2019
  2. IL-0454 Baemin billing servers could not open new MariaDB connections at the start of a card-discount event
    Woowa Brothers2016