Cases8 cases

3 to 7 of 8

1. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitApplication database

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
2. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy

   UK Ministry of Justice2023

   Symptoms

   At 09:14 on 4 August 2023, users of the Cloud Platform reported in the #ask-cloud-platform Slack channel that Kibana was missing their application logs for long periods.

   Monitoring outage

   Trigger

   The newer Fluent Bit version in use had changed its chunk drop strategy; the fix that followed handled memory buffer overflow.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agent

   Contributing factor

   No alert on limitMonitoring blind spot

   Detection

   Users reported long periods of missing logs in Kibana in the #ask-cloud-platform Slack channel at 09:14; the incident was declared at 10:09.

   Remediation

   • configured Fluent Bit to handle memory buffer overflow by writing buffered logs to the file system
   • made Fluent Bit flush logs in smaller chunks

   Action items

   • Push notifications from the logging clusters to #lower-priority-alerts (#4704)
   • Add an integration test that checks logs are being sent to the logging cluster

   View case
3. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes

   Reddit2023

   Symptoms

   About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

   Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag

   Trigger

   An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

   Root cause

   Breaking change in an upgrade

   Component

   CalicoKubernetes API serverAdmission webhooketcd

   Contributing factor

   Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure

   Detection

   The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

   Remediation

   • deleted the Open Policy Agent webhook configurations to stop API server timeouts
   • terminated all worker nodes of the cluster
   • restored the control plane from the pre-upgrade backup onto the first control plane node
   • switched off the cluster autoscaler until the control plane was stable
   • generated new TLS certificates so replacement control plane nodes could join etcd
   • brought traffic back in steps from 1% to 100%
   • restarted legacy services that had been stopped by hand

   Action items

   • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
   • Standardise components across clusters and replace bespoke configurations
   • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
   • Add pre-production clusters that can be tested with confidence
   • Improve the backup restore process and make cluster upgrades safe

   Lessons

   Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

   View case
4. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch

   PrometheusKube2020

   Symptoms

   On a Tuesday morning a developer reported that some production logs were missing. Only the logs from a single node were gone; the other nodes kept shipping normally through fluent-bit, the log shipper the team had run for more than three years.

   Monitoring outage

   Root cause

   Software defect

   Component

   Telemetry agentElasticsearch or OpenSearch

   Contributing factor

   Monitoring blind spotUnpatched known bug

   Detection

   A developer reported missing production logs; no alert fired because fluent-bit's metrics showed no errors.

   Remediation

   • added CPU to the Elasticsearch virtual machines, which cleared the write queue rejections
   • rolled out Fluentd one node at a time, using node affinity to place Fluentd and anti-affinity to remove fluent-bit
   • lowered Fluentd's flush_thread_count to 1 and set flush_interval to 5 seconds so the backlog of old logs did not overflow the buffer
   • enabled Fluentd's Position DB so already-ingested data was skipped

   Action items

   • Build monitoring for Fluentd on its Prometheus integration, including a throughput alert with separate day and night thresholds or a black-box log check

   Lessons

   A log shipper can fail without an error message and with metrics that simply stop moving, so error-based alerts never fire; throughput alerts or end-to-end checks can catch it. Open source software can break in ways nobody expects, and sometimes the way out is to switch to a different shipper fast, node by node. A new shipper that starts from scratch replays every old log on the node and needs to be slowed down.

   View case
5. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case

By root cause5