Incident Library
Applied filters DomainKubernetesOrganisationUK Ministry of JusticeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases17 cases

17 to 17 of 17

  1. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
    Symptoms

    At 10:43 on 14 August 2020 two of the six ingress controller pods in the cluster were in a crash loop. Restarting those pods did not fix them.

    Container crash loop
    Root cause
    Capacity exhaustion
    Component
    Ingress controller
    Contributing factor
    Harmful manual actionSingle point of failure
    Detection

    Two of the six ingress controller pods were seen crash-looping at 10:43; the incident was declared at 11:01.

    Remediation
    • restarted the leader ingress controller pod so the other replicas could connect and get the latest nginx configuration
    View case
Check your cluster for this

These 17 cases involve Ingress controller and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause7

Breaking change in an upgrade2 cases

Component
Ingress controller2Admission webhook1Web application firewall rules1
Contributing factor
Unrepresentative staging2
Organisation
UK Ministry of Justice2
  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  2. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021

Faulty deployment2 cases

Component
Admission webhook1Cluster autoscaler1Descheduler1Ingress controller1Node pool1
Contributing factor
Unreviewed defaults1No alert on limit1No headroom1Single point of failure1
Organisation
UK Ministry of Justice2
  1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  2. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020

Operator error3 cases

Component
CI system2Cloud load balancer1Credential broker for pods1external-dns1Ingress controller1Node launch template1Node pool1Prometheus1Container registry and its credentials1Terraform1VPC route tables and firewall rules1
Contributing factor
No safeguard on destructive change2Credential in configuration1Shared credential1
Organisation
UK Ministry of Justice3
  1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
  2. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023
  3. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020

Capacity exhaustion5 cases

Component
Prometheus2Telemetry agent2Elasticsearch or OpenSearch1Ingress controller1Container memory limit1Node pool1
Contributing factor
No alert on limit3Unreviewed defaults1Unknown limit1Harmful manual action1Monitoring blind spot1Single point of failure1
Organisation
UK Ministry of Justice5
  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  2. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  3. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  4. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
  5. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020

unknown1 case

Component
Prometheus
Contributing factor
–
Organisation
UK Ministry of Justice
  1. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics
    UK Ministry of Justice2023

Configuration change3 cases

Component
Node pool3AWS VPC CNI2CoreDNS1Node launch template1Node root disk1Terraform1
Contributing factor
No configuration tests2Excessive blast radius1No alert on limit1No runbook1Missed in review1
Organisation
UK Ministry of Justice3
  1. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
  2. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating
    UK Ministry of Justice2023
  3. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023

Cloud provider outage1 case

Component
Cloud load balancerIngress controller
Contributing factor
–
Organisation
UK Ministry of Justice
  1. IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone
    UK Ministry of Justice2022