Cases3 cases

1 to 3 of 3

1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space

   UK Ministry of Justice2024

   Symptoms

   Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.

   Monitoring outage

   Trigger

   OpenSearch ran out of disk space and index writes were blocked.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentElasticsearch or OpenSearch

   Contributing factor

   No alert on limit

   Detection

   A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.

   Remediation

   • restarted all fluent-bit containers (logs flowed for a few minutes, then the error came back)
   • increased OpenSearch disk space from 8000 to 12000
   • configured fluent-bit to stop sending logs to OpenSearch while the root cause was investigated

   Action items

   • Look into index dating issues in OpenSearch and Elasticsearch
   • Add high-priority alerts for Elasticsearch and OpenSearch
   • Re-introduce OpenSearch into live logging
   • Investigate the fluent-bit "failed to flush chunk" error

   View case
2. IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0

   Palark2021

   Symptoms

   After the last old Elasticsearch pod, elasticsearch-0, was deleted and came back on a new, empty persistent volume, `_cat/nodes` run inside that pod listed a single node: elasticsearch-0 itself, marked as master. The same command run inside elasticsearch-1 listed the other five nodes, elasticsearch-1 to elasticsearch-5, with elasticsearch-2 as master.

   Split brain

   Trigger

   Deleting the elasticsearch-0 pod and its PVC so that it restarted on an empty volume of the new StorageClass.

   Root cause

   Operator error

   Component

   Elasticsearch or OpenSearchPersistent volumes and stateful setsHelm

   Contributing factor

   Unverified assumption

   Detection

   The engineer running the migration checked `_cat/nodes` after elasticsearch-0 restarted and saw it alone in a cluster of its own.

   Remediation

   • scaled the three deployments of workers writing to Elasticsearch down to zero
   • listed elasticsearch-0, elasticsearch-1 and elasticsearch-2 under initial_master_nodes in the elasticsearch ConfigMap and removed the cluster.initial_master_nodes environment variable from the StatefulSet
   • saved and deleted the elasticsearch discovery Service and deleted the PVC of elasticsearch-0 while the rolling update ran
   • set initial_master_nodes to elasticsearch-3, elasticsearch-4 and elasticsearch-5, created an empty PVC for elasticsearch-0 and deleted all six pods
   • recreated the discovery Service from the saved manifest, after which the six nodes formed one cluster with every index green

   Action items

   • Restore the original initial_master_nodes values in the ConfigMap and restart all pods again
   • Move all shards to nodes 0, 1 and 2 and scale the cluster down from six nodes to three
   • Commit all the manual changes to the repository

   Lessons

   A setting that only matters when a cluster first starts, such as Elasticsearch's cluster.initial_master_nodes, can sit in a Helm chart template rather than in the ConfigMap that was reviewed. When a pod named in it starts with an empty data directory, it can bootstrap a new cluster beside the existing one. The author's conclusion is that a migration plan like this one breaks on any surprise and needs a plan B ready. Before running it in a more important environment, rehearse it in a test environment configured like production, and either schedule downtime or move the load to another cluster: stop the writers, take a fresh backup, then move the data.

   View case
3. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch

   PrometheusKube2020

   Symptoms

   On a Tuesday morning a developer reported that some production logs were missing. Only the logs from a single node were gone; the other nodes kept shipping normally through fluent-bit, the log shipper the team had run for more than three years.

   Monitoring outage

   Root cause

   Software defect

   Component

   Telemetry agentElasticsearch or OpenSearch

   Contributing factor

   Monitoring blind spotUnpatched known bug

   Detection

   A developer reported missing production logs; no alert fired because fluent-bit's metrics showed no errors.

   Remediation

   • added CPU to the Elasticsearch virtual machines, which cleared the write queue rejections
   • rolled out Fluentd one node at a time, using node affinity to place Fluentd and anti-affinity to remove fluent-bit
   • lowered Fluentd's flush_thread_count to 1 and set flush_interval to 5 seconds so the backlog of old logs did not overflow the buffer
   • enabled Fluentd's Position DB so already-ingested data was skipped

   Action items

   • Build monitoring for Fluentd on its Prometheus integration, including a throughput alert with separate day and night thresholds or a black-box log check

   Lessons

   A log shipper can fail without an error message and with metrics that simply stop moving, so error-based alerts never fire; throughput alerts or end-to-end checks can catch it. Open source software can break in ways nobody expects, and sometimes the way out is to switch to a different shipper fast, node by node. A new shipper that starts from scratch replays every old log on the node and needs to be slowed down.

   View case

By root cause3