Cases3 cases

1 to 3 of 3

1. IL-0455 Calls from an AWS service through NAT Gateway to an on-premises service returned 5xx errors every busy weekend

   Woowa Brothers2020

   Symptoms

   5xx errors rose on API requests from service A, running in AWS, to service B, running on premises, and user requests failed. At the same time the L4 session count of service B went up.

   High error rateOutbound connection exhaustion

   Trigger

   High weekend traffic from service A, which called service B without a connection pool, through the NAT gateways.

   Root cause

   Capacity exhaustion

   Component

   NAT gatewayOn-premises load balancer

   Contributing factor

   Unknown limit

   Detection

   Rising 5xx errors on service A and a rising L4 session count on service B.

   Remediation

   • Added service A instances, which lowered but did not end the errors
   • Replaced the service A instances, which brought errors and the L4 session count back to normal

   Action items

   • Switch service A to a connection pool
   • Find the services that use NAT Gateway between AWS and the IDC and switch them to connection pools

   Lessons

   The fix is often easy to find once the cause is analysed, but without confirming the root cause a service keeps running with the risk still in it, so Woowa Brothers treats finding the root cause as essential for preventing recurrence, alongside resolving the failure first.

   View case
2. IL-0573 A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stopped

   Woowa Brothers2016

   Symptoms

   At 15:00 on 27 December 2016, while a developer was working on a payment system module, nine main tables of the billing database were dropped. The monitoring system turned red, failure alerts poured in, and payments in 배달의민족 (Baemin) stopped for a while.

   Data lossService outage

   Trigger

   While developing a payment system module, the developer who wrote the post dropped nine main tables of the billing database by mistake.

   Root cause

   Operator error

   Component

   DatabaseDatabase backups and snapshots

   Contributing factor

   –

   Detection

   The monitoring system turned red and failure alerts fired.

   Remediation

   • Pointed the application at a temporary database, which kept the outage short
   • Restored the dropped tables from the 6:00 mysqldump full backup
   • Replayed the binary log from 6:00 to 15:00 with mysqlbinlog, filtered to the nine tables and stopping before the DROP TABLE

   Action items

   • Replace mysqldump backups of the billing database with full backups taken by XtraBackup

   Lessons

   There are many points where such failures can be managed: separate database accounts for the service and for administration, check once more before running a query, and guard against rm -rf. Without the full backup and the binary log the data could not have been recovered, so every team should check that it has a full recovery point for a failure.

   View case
3. IL-0454 Baemin billing servers could not open new MariaDB connections at the start of a card-discount event

   Woowa Brothers2016

   Symptoms

   At 18:00, when a card-company event giving 3,000 won off to the first 1,000 customers started, the Pinpoint graph of the Baemin billing system turned red shortly afterwards: 140 of 14,000 requests failed in about one minute. Most errors looked like requests from the billing system that the payment gateway (PG) did not receive because of a network problem, but other errors pointed at the billing system itself.

   High error rateIntermittent connection failures

   Trigger

   A card-company discount event for the first 1,000 customers started at 18:00 and sent about 14,000 requests to the billing system in about one minute.

   Root cause

   Software defect

   Component

   DatabaseClient-side DNS resolver and cache

   Contributing factor

   Unpatched known bug

   Detection

   The team was watching NewRelic and Pinpoint during the 18:00 event, and the Pinpoint graph turned red shortly after the event began.

   Remediation

   • Changed the Gradle settings to use mariadb-connector-j 1.5.5 instead of 1.4.6

   Lessons

   A new DB connection failed because the driver did a DNS lookup of the local host name that took about 6 seconds, longer than the server's 5-second connect_timeout. Other ways to avoid it are to use localhost as the host name or to register the IP address and host name in /etc/hosts so the address is returned first. Above all, the author recommends keeping libraries on recent versions.

   View case

By root cause3