Incident Library
Applied filters Root causeCapacity exhaustionFactorNo alert on limitOrganisationUK Ministry of JusticeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
    Symptoms

    Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.

    Monitoring outage
    Trigger

    OpenSearch ran out of disk space and index writes were blocked.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agentElasticsearch or OpenSearch
    Contributing factor
    No alert on limit
    Detection

    A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.

    Remediation
    • restarted all fluent-bit containers (logs flowed for a few minutes, then the error came back)
    • increased OpenSearch disk space from 8000 to 12000
    • configured fluent-bit to stop sending logs to OpenSearch while the root cause was investigated
    Action items
    • Look into index dating issues in OpenSearch and Elasticsearch
    • Add high-priority alerts for Elasticsearch and OpenSearch
    • Re-introduce OpenSearch into live logging
    • Investigate the fluent-bit "failed to flush chunk" error
    View case
  2. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
    Symptoms

    At 09:14 on 4 August 2023, users of the Cloud Platform reported in the #ask-cloud-platform Slack channel that Kibana was missing their application logs for long periods.

    Monitoring outage
    Trigger

    The newer Fluent Bit version in use had changed its chunk drop strategy; the fix that followed handled memory buffer overflow.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agent
    Contributing factor
    No alert on limitMonitoring blind spot
    Detection

    Users reported long periods of missing logs in Kibana in the #ask-cloud-platform Slack channel at 09:14; the incident was declared at 10:09.

    Remediation
    • configured Fluent Bit to handle memory buffer overflow by writing buffered logs to the file system
    • made Fluent Bit flush logs in smaller chunks
    Action items
    • Push notifications from the logging clusters to #lower-priority-alerts (#4704)
    • Add an integration test that checks logs are being sent to the logging cluster
    View case
  3. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
    Symptoms

    At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.

    Monitoring outageOOMKilledNode NotReady
    Trigger

    The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.

    Root cause
    Capacity exhaustion
    Component
    PrometheusNode poolContainer memory limit
    Contributing factor
    No alert on limit
    Detection

    A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.

    Remediation
    • drained the monitoring node, moving Prometheus to another monitoring node
    • changed the node instance size to r6i.4xlarge
    • raised the Prometheus memory limit to 100Gi
    • set the Prometheus container limits to 12 CPU cores and 110Gi of memory
    Action items
    • Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)
    View case
Check your cluster for this

These 3 cases involve Telemetry agent and Elasticsearch or OpenSearch. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion3 cases

Component
Telemetry agent2Elasticsearch or OpenSearch1Container memory limit1Node pool1Prometheus1
Contributing factor
No alert on limit3Monitoring blind spot1
Organisation
UK Ministry of Justice3
  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  2. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  3. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023