Incident Library
Applied filters ComponentTelemetry agentSymptomMonitoring outageFactorNo alert on limitClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
    Symptoms

    Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.

    Monitoring outage
    Trigger

    OpenSearch ran out of disk space and index writes were blocked.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agentElasticsearch or OpenSearch
    Contributing factor
    No alert on limit
    Detection

    A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.

    Remediation
    • restarted all fluent-bit containers (logs flowed for a few minutes, then the error came back)
    • increased OpenSearch disk space from 8000 to 12000
    • configured fluent-bit to stop sending logs to OpenSearch while the root cause was investigated
    Action items
    • Look into index dating issues in OpenSearch and Elasticsearch
    • Add high-priority alerts for Elasticsearch and OpenSearch
    • Re-introduce OpenSearch into live logging
    • Investigate the fluent-bit "failed to flush chunk" error
    View case
  2. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
    Symptoms

    At 09:14 on 4 August 2023, users of the Cloud Platform reported in the #ask-cloud-platform Slack channel that Kibana was missing their application logs for long periods.

    Monitoring outage
    Trigger

    The newer Fluent Bit version in use had changed its chunk drop strategy; the fix that followed handled memory buffer overflow.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agent
    Contributing factor
    No alert on limitMonitoring blind spot
    Detection

    Users reported long periods of missing logs in Kibana in the #ask-cloud-platform Slack channel at 09:14; the incident was declared at 10:09.

    Remediation
    • configured Fluent Bit to handle memory buffer overflow by writing buffered logs to the file system
    • made Fluent Bit flush logs in smaller chunks
    Action items
    • Push notifications from the logging clusters to #lower-priority-alerts (#4704)
    • Add an integration test that checks logs are being sent to the logging cluster
    View case
Check your cluster for this

These 2 cases involve Telemetry agent and Elasticsearch or OpenSearch. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Telemetry agent2Elasticsearch or OpenSearch1
Contributing factor
No alert on limit2Monitoring blind spot1
Organisation
UK Ministry of Justice2
  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  2. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023