Cases3 cases

1 to 3 of 3

1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space

   UK Ministry of Justice2024

   Symptoms

   Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.

   Monitoring outage

   Trigger

   OpenSearch ran out of disk space and index writes were blocked.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentElasticsearch or OpenSearch

   Contributing factor

   No alert on limit

   Detection

   A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.

   Remediation

   • restarted all fluent-bit containers (logs flowed for a few minutes, then the error came back)
   • increased OpenSearch disk space from 8000 to 12000
   • configured fluent-bit to stop sending logs to OpenSearch while the root cause was investigated

   Action items

   • Look into index dating issues in OpenSearch and Elasticsearch
   • Add high-priority alerts for Elasticsearch and OpenSearch
   • Re-introduce OpenSearch into live logging
   • Investigate the fluent-bit "failed to flush chunk" error

   View case
2. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitApplication database

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
3. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy

   UK Ministry of Justice2023

   Symptoms

   At 09:14 on 4 August 2023, users of the Cloud Platform reported in the #ask-cloud-platform Slack channel that Kibana was missing their application logs for long periods.

   Monitoring outage

   Trigger

   The newer Fluent Bit version in use had changed its chunk drop strategy; the fix that followed handled memory buffer overflow.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agent

   Contributing factor

   No alert on limitMonitoring blind spot

   Detection

   Users reported long periods of missing logs in Kibana in the #ask-cloud-platform Slack channel at 09:14; the incident was declared at 10:09.

   Remediation

   • configured Fluent Bit to handle memory buffer overflow by writing buffered logs to the file system
   • made Fluent Bit flush logs in smaller chunks

   Action items

   • Push notifications from the logging clusters to #lower-priority-alerts (#4704)
   • Add an integration test that checks logs are being sent to the logging cluster

   View case

By root cause1