IL-0580 Recruit · 2019-12-18 (published)networkingaws

Deploys on a ConcourseCI pipeline failed intermittently with DNS lookup timeouts when pulling images from Amazon ECR

Recruit Technologies runs the product on containers and uses ConcourseCI, set up with docker-compose on EC2, for CI/CD. The engineer found a Go and Docker issue that described the same symptom. The Docker in the Concourse job was 17.03.2-ce built with go1.7.5, the Go version named in the issue, which traced the failure to a bug in the Go standard library that Docker was built with. Fixing that meant updating Docker, either by upgrading Concourse, which needed verification and could not be done quickly, or by rewriting the resource-type.

The answers seen in the CI job were larger than the same answers seen from a local machine. As an interim fix, the engineer changed the nameserver in resolv.conf on the CI worker to a different DNS cache server, and deploys worked again. The engineer then looked into why the sizes differed: the answers in the CI job were larger because they were not compressed. DNS has a message compression feature (RFC section 4.1.4. Message compression). Packet captures on each network showed that the Concourse worker had its own DNS proxy, which forwarded responses without compressing them. The engineer then filed an issue and a pull request to Concourse. In short, the size of ECR's DNS responses varied, the Concourse internal DNS proxy forwarded them uncompressed, and together they hit the Go bug.

It took more than a month from the failure to the pull request. Much of that time went to finding which DNS server expanded the compressed messages: the engineer first suspected Docker's internal DNS, read the libnetwork code, built patched Docker and DNS libraries in a Vagrant reproduction, and analysed the iptables rules that Concourse and Docker create, all of which turned out to be beside the point.

Severity: –
Scope: Deploys through the ConcourseCI pipeline failed on some days
Users affected: –
Region: ap-northeast-1
Time to detect: –
Time to resolve: –

Symptoms

When an engineer at Recruit Technologies pressed the deploy button, Docker running in a ConcourseCI job failed to pull an image from Amazon ECR that docker build uses as cache. The error was: Error response from daemon: Get https://xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/v2/: dial tcp: lookup xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com on 172.18.0.3:53: read udp 10.254.0.122:48372->172.18.0.3:53: i/o timeout

The deploy flow was git clone, docker build, docker push, then rewriting and deploying the Kubernetes manifest. The error had never appeared since the service was released, and no configuration had changed recently. Resolving the ECR name from a local machine and from a CI job worked, and the CI also resolved the name of GitHub Enterprise without trouble.

The engineer first took the error for temporary network instability, perhaps in Route53 or the VPC. A retry that night also failed. A retry the next day succeeded, but the same error came back a few days later, shortly before a release whose date could not move.

Comparing past results, the ECR name, a CNAME to an Elastic Load Balancer, resolved to 2 A records on days that worked and to 4 or more on days that failed. dig from a CI job showed MSG SIZE rcvd: 580, while the same query from a local machine showed MSG SIZE rcvd: 218.

Symptom DNS resolution failures Image pull failures

Root cause

Root cause Software defect

Component CI system Container registry and its credentials Client-side DNS resolver and cache

Contributing factor Unpatched known bug

Trigger: The Elastic Load Balancer behind the ECR endpoint returned more A records, as an ELB does when its units grow with request volume, which made the DNS responses larger.

Detection

The deploy failed with the DNS lookup timeout in the CI job when an engineer pressed the deploy button.

Remediation

Changed the nameserver in resolv.conf on the CI worker to another DNS cache server as an interim fix

Action items

Filed an issue and a pull request to Concourse to fix its DNS proxy forwarding uncompressed responses

Lessons

–

Sources

https://techblog.recruit.co.jp/article-204/ · blog · archived copy

Related cases

IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolverToyota Connected · 2018
same root cause · shared Client-side DNS resolver and cache
IL-0454 Baemin billing servers could not open new MariaDB connections at the start of a card-discount eventWoowa Brothers · 2016
same root cause · shared Client-side DNS resolver and cache

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CI system, Container registry and its credentials, Client-side DNS resolver and cache among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0580
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0580/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.