Incident Library

IL-0454 Woowa Brothers · 2016-12-08databasenetworking

Baemin billing servers could not open new MariaDB connections at the start of a card-discount event

The Baemin billing system passes users' payment requests to payment gateways and hands the responses to the order system. It runs on 7 Spring Boot application servers with MariaDB, using mariadb-connector-j and a JDBC connection pool. The team used io.spring.dependency-management 0.5.6, which picks mariadb-java-client 1.4.6 automatically. After a senior engineer on the team changed the Gradle settings to the latest mariadb-connector-j, 1.5.5, the problem no longer reproduced.

To learn why, the team went through the driver's changelog and found Jira issue CONJ-360: DriverManager.getConnection() threw SQLNonTransientConnectionException when creating a new connection, the problem disappeared with the MySQL connector, and it seemed to come from ManagementFactory.getRuntimeMXBean().getName(), which fetches the PID. Before the fix, SendHandshakeResponsePacket.java sent pid@hostname as part of the authentication data and got the host name with InetAddress.getLocalHost().getHostName(). Unless the host name is 'localhost', getLocalHost() resolves it through the name service (DNSNameService's lookupAllHostAddr()), caching the answer for 5 seconds. On the billing application servers that lookup took about 6 seconds, while the billing database's connect_timeout was 5 seconds, so the database's connect_timeout fired before the new connection finished its handshake. The fix, which returns only the PID through JNA, was released in October 2016 as version 1.5.4.

Severity
–
Scope
Payment requests through the Baemin billing system at the start of the event: 140 of 14,000 requests failed in about one minute
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

At 18:00, when a card-company event giving 3,000 won off to the first 1,000 customers started, the Pinpoint graph of the Baemin billing system turned red shortly afterwards: 140 of 14,000 requests failed in about one minute. Most errors looked like requests from the billing system that the payment gateway (PG) did not receive because of a network problem, but other errors pointed at the billing system itself.

The application threw DataAccessResourceFailureException: Unable to acquire JDBC Connection; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection, caused by java.sql.SQLNonTransientConnectionException: Could not connect to xxx.xxx.xxx.xxx unexpected end of stream, read 0 bytes from 4. The database error log showed 2016-12-08 18:00:55 xxxx [Warning] Aborted connection 37170 to db: ‘unconnected’ user: ‘unauthenticated’ host: ‘xxx.xxx.xxx.xxx’ (Unknown error).

The team first suspected the connection pool settings but found nothing wrong, and the errors stopped on their own over time. Reading the messages as new DB connections failing at the authentication step, the team searched for 'mysql unauthenticated user'. Most results pointed to the --skip-name-resolve option, which stops MySQL and MariaDB from looking up the client's host name from its IP address on each new connection, but the billing database already had skip_name_resolve set to ON.

Symptom High error rateIntermittent connection failures

Root cause

Root cause Software defect

Component DatabaseClient-side DNS resolver and cache
Contributing factor Unpatched known bug
Trigger
A card-company discount event for the first 1,000 customers started at 18:00 and sent about 14,000 requests to the billing system in about one minute.

Detection

The team was watching NewRelic and Pinpoint during the 18:00 event, and the Pinpoint graph turned red shortly after the event began.

Remediation

Action items

–

Lessons

A new DB connection failed because the driver did a DNS lookup of the local host name that took about 6 seconds, longer than the server's 5-second connect_timeout. Other ways to avoid it are to use localhost as the host name or to register the IP address and host name in /etc/hosts so the address is returned first. Above all, the author recommends keeping libraries on recent versions.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Client-side DNS resolver and cache among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.