Cases1 case

1 to 1 of 1

1. IL-0454 Baemin billing servers could not open new MariaDB connections at the start of a card-discount event

   Woowa Brothers2016

   Symptoms

   At 18:00, when a card-company event giving 3,000 won off to the first 1,000 customers started, the Pinpoint graph of the Baemin billing system turned red shortly afterwards: 140 of 14,000 requests failed in about one minute. Most errors looked like requests from the billing system that the payment gateway (PG) did not receive because of a network problem, but other errors pointed at the billing system itself.

   High error rateIntermittent connection failures

   Trigger

   A card-company discount event for the first 1,000 customers started at 18:00 and sent about 14,000 requests to the billing system in about one minute.

   Root cause

   Software defect

   Component

   DatabaseClient-side DNS resolver and cache

   Contributing factor

   Unpatched known bug

   Detection

   The team was watching NewRelic and Pinpoint during the 18:00 event, and the Pinpoint graph turned red shortly after the event began.

   Remediation

   • Changed the Gradle settings to use mariadb-connector-j 1.5.5 instead of 1.4.6

   Lessons

   A new DB connection failed because the driver did a DNS lookup of the local host name that took about 6 seconds, longer than the server's 5-second connect_timeout. Other ways to avoid it are to use localhost as the host name or to register the IP address and host name in /etc/hosts so the address is returned first. Above all, the author recommends keeping libraries on recent versions.

   View case

By root cause1