IL-0364 UK Ministry of Justice · 2022-07-11awskubernetesnetworking
A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone
After an AWS incident on 10 July 2022, the health checking subsystem of the NLB in front of the Cloud Platform's 'default' ingress controllers lost track of the targets in one Availability Zone. According to AWS, a latent software race condition was triggered when some health checking instances were restarted; the subsystem then returned no health status for that zone, did not detect some targets there as unhealthy, and clients connecting to one of the NLB's Elastic IPs timed out. The problem was first detected at 09:33 on 11 July and declared an incident at 10:11. AWS restarted the health checking subsystem, which refreshed its list of targets and recovered the NLB in the affected zone, and the incident was resolved at 16:07, with a time to repair of 6h 27m.
- Severity
- –
- Scope
- Slow performance for 25% of ingress traffic through the 'default' ingress controllers
- Users affected
- –
- Region
- one Availability Zone
- Time to detect
- –
- Time to resolve
- 6h 27m
Symptoms
On 11 July 2022 users reported in the #ask-cloud-platform Slack channel that their applications on the Ministry of Justice Cloud Platform were slow some of the time. About 25% of ingress traffic was affected: after an AWS incident the day before, one of the three network interfaces of the 'default' ingress controllers performed slowly, and clients timed out when they tried to connect to one of the Elastic IPs of the Network Load Balancer (NLB) in front of them.
Root cause
Root cause Cloud provider outage
- Trigger
- Some instances of the AWS NLB health checking subsystem were restarted, which set off a latent software race condition in that subsystem.
Detection
Users reported slow performance of their applications in the #ask-cloud-platform Slack channel.
Remediation
- AWS restarted the NLB health checking subsystem, which refreshed its list of targets and recovered the NLB in the affected Availability Zone
Action items
- Raised mitigation tickets after a post-incident review, filed as post-aws-incident issues in the ministryofjustice/cloud-platform repository
Lessons
–
Sources
- https://runbooks.cloud-platform.service.justice.gov.uk/incidents/2022-07-11-slow-performance-for-ingress-traffic.html · runbook · archived copy
References
Related cases
IL-0438Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failedANDPAD · 2026same root cause · shared Ingress controller, Cloud load balancer
IL-0463Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachableHarness · 2026same root cause · shared Cloud load balancer
IL-0542Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE clusterTwingate · 2025same root cause · shared Cloud load balancer
IL-0446About 20% of requests to an ITANDI ECS service got ALB 5XX errors, apparently from one zone, right after a deployITANDI · 2020same root cause · shared Cloud load balancer
IL-0014A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019same root cause · shared Ingress controller
IL-0494Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hungNetlify · 2016same root cause · shared Cloud load balancer
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0364
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0364/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.