Incident Library
Applied filters SymptomMonitoring outageOrganisationUK Ministry of JusticeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
    Symptoms

    At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

    Service outageResources deletedMonitoring outageStale endpoint routing
    Trigger

    A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

    Root cause
    Operator error
    Component
    Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus
    Contributing factor
    No safeguard on destructive change
    Detection

    The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

    Remediation
    • restored the default ingress controller together with external-dns and kiam, which it depends on
    • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
    • restored the remaining components with a targeted apply
    • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository
    View case
Check your cluster for this

These 6 cases involve Prometheus and Telemetry agent. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Capacity exhaustion4 cases

Component
Prometheus2Telemetry agent2Elasticsearch or OpenSearch1Container memory limit1Node pool1
Contributing factor
No alert on limit3Unreviewed defaults1Unknown limit1Monitoring blind spot1
Organisation
UK Ministry of Justice4
  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  2. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  3. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  4. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023

unknown1 case

Component
Prometheus
Contributing factor
–
Organisation
UK Ministry of Justice
  1. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics
    UK Ministry of Justice2023

Operator error1 case

Component
Cloud load balancerCredential broker for podsexternal-dnsIngress controllerPrometheus
Contributing factor
No safeguard on destructive change
Organisation
UK Ministry of Justice
  1. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020