Cases2 cases

1 to 2 of 2

1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space

   UK Ministry of Justice2024

   Symptoms

   Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.

   Monitoring outage

   Trigger

   OpenSearch ran out of disk space and index writes were blocked.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentElasticsearch or OpenSearch

   Contributing factor

   No alert on limit

   Detection

   A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.

   Remediation

   • restarted all fluent-bit containers (logs flowed for a few minutes, then the error came back)
   • increased OpenSearch disk space from 8000 to 12000
   • configured fluent-bit to stop sending logs to OpenSearch while the root cause was investigated

   Action items

   • Look into index dating issues in OpenSearch and Elasticsearch
   • Add high-priority alerts for Elasticsearch and OpenSearch
   • Re-introduce OpenSearch into live logging
   • Investigate the fluent-bit "failed to flush chunk" error

   View case
2. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch

   PrometheusKube2020

   Symptoms

   On a Tuesday morning a developer reported that some production logs were missing. Only the logs from a single node were gone; the other nodes kept shipping normally through fluent-bit, the log shipper the team had run for more than three years.

   Monitoring outage

   Root cause

   Software defect

   Component

   Telemetry agentElasticsearch or OpenSearch

   Contributing factor

   Monitoring blind spotUnpatched known bug

   Detection

   A developer reported missing production logs; no alert fired because fluent-bit's metrics showed no errors.

   Remediation

   • added CPU to the Elasticsearch virtual machines, which cleared the write queue rejections
   • rolled out Fluentd one node at a time, using node affinity to place Fluentd and anti-affinity to remove fluent-bit
   • lowered Fluentd's flush_thread_count to 1 and set flush_interval to 5 seconds so the backlog of old logs did not overflow the buffer
   • enabled Fluentd's Position DB so already-ingested data was skipped

   Action items

   • Build monitoring for Fluentd on its Prometheus integration, including a throughput alert with separate day and night thresholds or a black-box log check

   Lessons

   A log shipper can fail without an error message and with metrics that simply stop moving, so error-based alerts never fire; throughput alerts or end-to-end checks can catch it. Open source software can break in ways nobody expects, and sometimes the way out is to switch to a different shipper fast, node by node. A new shipper that starts from scratch replays every old log on the node and needs to be slowed down.

   View case

By root cause2