Incident Library
Applied filters ComponentCloud load balancerSymptomIntermittent timeoutsOrganisationUK Ministry of JusticeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone
    UK Ministry of Justice2022
    Symptoms

    On 11 July 2022 users reported in the #ask-cloud-platform Slack channel that their applications on the Ministry of Justice Cloud Platform were slow some of the time. About 25% of ingress traffic was affected: after an AWS incident the day before, one of the three network interfaces of the 'default' ingress controllers performed slowly, and clients timed out when they tried to connect to one of the Elastic IPs of the Network Load Balancer (NLB) in front of them.

    High latencyIntermittent timeouts
    Trigger

    Some instances of the AWS NLB health checking subsystem were restarted, which set off a latent software race condition in that subsystem.

    Root cause
    Cloud provider outage
    Component
    Cloud load balancerIngress controller
    Contributing factor
    –
    Detection

    Users reported slow performance of their applications in the #ask-cloud-platform Slack channel.

    Remediation
    • AWS restarted the NLB health checking subsystem, which refreshed its list of targets and recovered the NLB in the affected Availability Zone
    Action items
    • Raised mitigation tickets after a post-incident review, filed as post-aws-incident issues in the ministryofjustice/cloud-platform repository
    View case
Check your cluster for this

This case involves Cloud load balancer and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
Cloud load balancerIngress controller
Contributing factor
–
Organisation
UK Ministry of Justice
  1. IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone
    UK Ministry of Justice2022