Cases1 case

1 to 1 of 1

1. IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone

   UK Ministry of Justice2022

   Symptoms

   On 11 July 2022 users reported in the #ask-cloud-platform Slack channel that their applications on the Ministry of Justice Cloud Platform were slow some of the time. About 25% of ingress traffic was affected: after an AWS incident the day before, one of the three network interfaces of the 'default' ingress controllers performed slowly, and clients timed out when they tried to connect to one of the Elastic IPs of the Network Load Balancer (NLB) in front of them.

   High latencyIntermittent timeouts

   Trigger

   Some instances of the AWS NLB health checking subsystem were restarted, which set off a latent software race condition in that subsystem.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerIngress controller

   Contributing factor

   –

   Detection

   Users reported slow performance of their applications in the #ask-cloud-platform Slack channel.

   Remediation

   • AWS restarted the NLB health checking subsystem, which refreshed its list of targets and recovered the NLB in the affected Availability Zone

   Action items

   • Raised mitigation tickets after a post-incident review, filed as post-aws-incident issues in the ministryofjustice/cloud-platform repository

   View case

By root cause1