IL-0043 UK Ministry of Justice · 2023-08-04kubernetesobservability
On the Ministry of Justice Cloud Platform, Fluent Bit ships application logs to a logging cluster that users read through Kibana. On the morning of 4 August 2023 users reported long periods of missing logs. The team traced it to a change in how the newer Fluent Bit version drops chunks. The platform lost application logs for a period of time.
Restarts, a newer Fluent Bit version and a bigger memory buffer were tried first. The fix was to handle buffer overflow by letting Fluent Bit write to the file system, merged and deployed to the live cluster on 9 August, and to flush logs in smaller chunks, added on 10 August. The incident was repaired at 12:28 and resolved at 14:47 on 10 August; the report gives the time to repair as 33h 14m and the time to resolve as 35h 33m.
At 09:14 on 4 August 2023, users of the Cloud Platform reported in the #ask-cloud-platform Slack channel that Kibana was missing their application logs for long periods.
The platform team began investigating at 10:03 and restarted the Fluent Bit pods; the incident was declared at 10:09. At 12:03 they found that the newer Fluent Bit version had changed its strategy for dropping chunks. The team moved to another Fluent Bit version at 16:00 the same day and enlarged the Fluent Bit memory buffer on 7 August; the repair came later, with changes to how Fluent Bit buffers and flushes logs.
Symptom Monitoring outage
Root cause Capacity exhaustion
Users reported long periods of missing logs in Kibana in the #ask-cloud-platform Slack channel at 09:14; the incident was declared at 10:09.
–
IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024same root cause · shared Telemetry agent
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Telemetry agent
IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023same root cause · shared Telemetry agent
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Telemetry agent
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Telemetry agent