Incident Library
Applied filters ComponentElasticsearch or OpenSearchSymptomMonitoring outageOrganisationUK Ministry of JusticeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
    Symptoms

    Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.

    Monitoring outage
    Trigger

    OpenSearch ran out of disk space and index writes were blocked.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agentElasticsearch or OpenSearch
    Contributing factor
    No alert on limit
    Detection

    A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.

    Remediation
    • restarted all fluent-bit containers (logs flowed for a few minutes, then the error came back)
    • increased OpenSearch disk space from 8000 to 12000
    • configured fluent-bit to stop sending logs to OpenSearch while the root cause was investigated
    Action items
    • Look into index dating issues in OpenSearch and Elasticsearch
    • Add high-priority alerts for Elasticsearch and OpenSearch
    • Re-introduce OpenSearch into live logging
    • Investigate the fluent-bit "failed to flush chunk" error
    View case
Check your cluster for this

This case involves Elasticsearch or OpenSearch and Telemetry agent. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Elasticsearch or OpenSearchTelemetry agent
Contributing factor
No alert on limit
Organisation
UK Ministry of Justice
  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024