Kubernetes · kubegap
Why didn't my Kubernetes alerts fire during an outage?
Usually because the alert rules watch causes, not symptoms. Rules written for CPU or restarts miss a failure nobody predicted, while a rule on error rate or latency catches any of them. The other common reasons are an Alertmanager route that drops alerts, rules that have fired for days, and monitoring that depends on the cluster it watches.
Cause alerts versus symptom alerts
Google's Site Reliability Engineering book, in its chapter on monitoring distributed systems, draws the line that most alerting setups miss: alert on symptoms, what users experience, and keep causes for the dashboards. A rule on a CoreDNS pod's memory is a cause alert; it fires only for the failure someone imagined when writing it. A rule on the share of DNS lookups that fail is a symptom alert; it fires for that failure and for every other one with the same effect.
Clusters set up from a Helm chart tend to arrive with many cause alerts and few symptom alerts, because the chart knows about pods and nodes and nothing about your users. An audit that looks for this treats missing or too few symptom-based rules as a critical finding, and failure modes without any alert as a separate high one. The public cases attached to that second check in the audit's check list are FREE NOW's 2019 postmortem on new workers unable to join the cluster, NU.nl's 2019 write-up of a cluster failure and Saltside's 2017 account of their first Kubernetes outage.
A related finding is cause alerts routed at page severity. Many critical alerts that wake someone at night without a user-visible effect train the on-call to ignore the page, which is the failure mode of the next section.
Alerts that have been firing for a week
The second reason an outage passes unnoticed is that an alert did fire, into a channel where alerts fire all the time. An alert that has been in the firing state for seven days is no longer information; it is background. An audit should count alerts firing continuously for more than three days as a problem of their own and alerts firing for seven days as alert fatigue; in a published k3s sample audit a CPU throttling alert had been firing for 3.1 days with nobody acting on it.
The companion check is the opposite case: rules that have never fired in 15 days. Some are correct and quiet. Others watch a metric that no longer exists after a chart upgrade, and they would stay quiet through the outage they were written for. Reading the list of never-fired rules against the metrics Prometheus actually holds is a one-hour job that finds them.
The alert fired and went nowhere
Prometheus evaluates the rule; Alertmanager decides where it goes. Three things in between can swallow it. The default receiver can be empty, so an alert that matches no route vanishes. A receiver can be configured but referenced by no route. And notification delivery can fail, which Alertmanager records in its own metrics and nobody reads. Each of these is a check of its own: an empty default receiver, receivers no route uses, notification failures, and alerts that keep arriving while nothing has been sent to any receiver for 24 hours.
A single Alertmanager with cluster mode off is a smaller version of the same problem. When it restarts during the outage, the alerts that were about to be sent are lost with it.
Monitoring that depends on what it monitors
Zalando's January 2019 DNS postmortem is the clearest public case. CoreDNS was repeatedly OOM-killed under a burst of lookups, and resolution failed across the cluster. Their monitoring agents ran inside the same cluster and relied on the DNS that was failing, so the response was slower than it should have been. The lesson they drew was to limit the blast radius of one application's DNS traffic, but the monitoring lesson is the general one: a paging path that needs the cluster to be healthy cannot tell you the cluster is not.
The same dependency hides in storage. A Prometheus or Loki pod with its data on emptyDir loses everything when it is rescheduled, which is exactly what happens on a bad day. An observability stack on emptyDir is a high finding in the same audit, and the k3s sample report named monitoring data held in emptyDir as one of the day's three problems. Prometheus memory near its limit or an OOMKilled history is another; the UK Ministry of Justice's Cloud Platform team published two 2023 incident reports of Prometheus going down or restarting on their live cluster, and the check list attaches both to that check.
Nothing watched the thing that broke
The last reason is the most ordinary. The broken component exported no metrics, or exported them and nothing scraped them. Ingress controller metrics missing from Prometheus means the latency, traffic and errors of every backend behind it are invisible at once. CoreDNS, kube-proxy, the EBS CSI driver, Karpenter, KEDA and cert-manager each expose metrics that a default install may not scrape, and each of them deserves a metrics-unreadable check of its own.
Then there are logs. Running pods with no logs in the log store, a log agent dropping lines, or a log store rejecting them for rate limits all mean the error you would have searched for was never kept. The UK Ministry of Justice's 2023 and 2024 reports on dropped logging and Elasticsearch logging failures, and PrometheusKube's 2020 account of switching log agents, are the cases the audit cites here.
What to check this week
- Count your symptom alerts. For each service users depend on, is there a rule on its error rate and its latency? If the answer is a cause alert on CPU, write the symptom rule.
- List alerts in the firing state now and sort by how long. Anything over three days is either a real problem nobody owns or a rule to delete.
- Read the Alertmanager configuration for an empty default receiver and for receivers no route reaches, then read its notification failure metrics.
- Find where Prometheus, Alertmanager and Loki keep their data, and whether a restart of each would lose it. Ask the same question of the path a page takes to reach a phone.
- List the components that expose metrics nobody scrapes, starting with the ingress controller and DNS.
kubegap runs these checks as part of its observability area, 64 checks in all, when you type /kubegap in Claude Code or $kubegap in Codex, and its report says which rules, receivers and stores it looked at. The public incidents behind the checks, Zalando's DNS outage and the Ministry of Justice's Prometheus incidents among them, are searchable by symptom at Incident Library, https://benchlet.com/incidentlib/app, each with a link to the original report.
Frequently asked questions
What is a symptom-based alert?
A rule on what users experience, such as the error rate or latency of a service, rather than on an internal cause such as CPU or restarts. Google's Site Reliability Engineering book recommends paging on symptoms and keeping cause signals for diagnosis, because a symptom rule catches failures nobody predicted.
Our alert fired but nobody was paged. Where do I look?
In Alertmanager, not Prometheus. Check that the alert's labels match a route, that the default receiver is not empty, that the receiver it reached is referenced by a route, and read Alertmanager's notification failure metrics for that receiver.
Is an alert that has fired for a week harmful?
Yes, in two ways. It hides new alerts in the same channel, and it teaches the on-call that the channel can be ignored. Either fix the condition or change the rule; leaving it firing is the choice that costs the next outage.
Should monitoring run outside the cluster?
At least the paging path should not depend on the cluster being healthy, and the data should not be on emptyDir. Zalando's 2019 postmortem is the public example of monitoring inside the cluster slowing the response to a DNS outage. A second Prometheus, a remote write target or a managed service are the usual answers.