Cases1 case

1 to 1 of 1

1. IL-0455 Calls from an AWS service through NAT Gateway to an on-premises service returned 5xx errors every busy weekend

   Woowa Brothers2020

   Symptoms

   5xx errors rose on API requests from service A, running in AWS, to service B, running on premises, and user requests failed. At the same time the L4 session count of service B went up.

   High error rateOutbound connection exhaustion

   Trigger

   High weekend traffic from service A, which called service B without a connection pool, through the NAT gateways.

   Root cause

   Capacity exhaustion

   Component

   NAT gatewayOn-premises load balancer

   Contributing factor

   Unknown limit

   Detection

   Rising 5xx errors on service A and a rising L4 session count on service B.

   Remediation

   • Added service A instances, which lowered but did not end the errors
   • Replaced the service A instances, which brought errors and the L4 session count back to normal

   Action items

   • Switch service A to a connection pool
   • Find the services that use NAT Gateway between AWS and the IDC and switch them to connection pools

   Lessons

   The fix is often easy to find once the cause is analysed, but without confirming the root cause a service keeps running with the risk still in it, so Woowa Brothers treats finding the root cause as essential for preventing recurrence, alongside resolving the failure first.

   View case

By root cause1