Cases2 cases

1 to 2 of 2

1. IL-0580 Deploys on a ConcourseCI pipeline failed intermittently with DNS lookup timeouts when pulling images from Amazon ECR

   Recruit2019

   Symptoms

   When an engineer at Recruit Technologies pressed the deploy button, Docker running in a ConcourseCI job failed to pull an image from Amazon ECR that docker build uses as cache. The error was: Error response from daemon: Get https://xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/v2/: dial tcp: lookup xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com on 172.18.0.3:53: read udp 10.254.0.122:48372->172.18.0.3:53: i/o timeout

   DNS resolution failuresImage pull failures

   Trigger

   The Elastic Load Balancer behind the ECR endpoint returned more A records, as an ELB does when its units grow with request volume, which made the DNS responses larger.

   Root cause

   Software defect

   Component

   CI systemContainer registry and its credentialsClient-side DNS resolver and cache

   Contributing factor

   Unpatched known bug

   Detection

   The deploy failed with the DNS lookup timeout in the CI job when an engineer pressed the deploy button.

   Remediation

   • Changed the nameserver in resolv.conf on the CI worker to another DNS cache server as an interim fix

   Action items

   • Filed an issue and a pull request to Concourse to fix its DNS proxy forwarding uncompressed responses

   View case
2. IL-0454 Baemin billing servers could not open new MariaDB connections at the start of a card-discount event

   Woowa Brothers2016

   Symptoms

   At 18:00, when a card-company event giving 3,000 won off to the first 1,000 customers started, the Pinpoint graph of the Baemin billing system turned red shortly afterwards: 140 of 14,000 requests failed in about one minute. Most errors looked like requests from the billing system that the payment gateway (PG) did not receive because of a network problem, but other errors pointed at the billing system itself.

   High error rateIntermittent connection failures

   Trigger

   A card-company discount event for the first 1,000 customers started at 18:00 and sent about 14,000 requests to the billing system in about one minute.

   Root cause

   Software defect

   Component

   DatabaseClient-side DNS resolver and cache

   Contributing factor

   Unpatched known bug

   Detection

   The team was watching NewRelic and Pinpoint during the 18:00 event, and the Pinpoint graph turned red shortly after the event began.

   Remediation

   • Changed the Gradle settings to use mariadb-connector-j 1.5.5 instead of 1.4.6

   Lessons

   A new DB connection failed because the driver did a DNS lookup of the local host name that took about 6 seconds, longer than the server's 5-second connect_timeout. Other ways to avoid it are to use localhost as the host name or to register the IP address and host name in /etc/hosts so the address is returned first. Above all, the author recommends keeping libraries on recent versions.

   View case

By root cause1