Kubernetes · kubegap
How do I audit a Kubernetes cluster I just inherited?
Start with what loses data or takes everything down at once, not with the dashboards. In the first week, confirm every production database has a backup you could restore, find out which pods share a node, list what is open to the internet, read the load balancer health checks, and check versions against their support dates.
Why the dashboards are not the audit
A cluster you inherit comes with the monitoring its previous owners set up, and that monitoring shows what they thought to watch. The gaps are elsewhere: a volume nobody snapshots, a PodDisruptionBudget nobody wrote, a listener someone opened for a test. None of those raise an alert, because nothing was configured to alert on an absence.
So the first-week audit is a read of the cluster's objects and the cloud account, not of the graphs. The five problems below are, in this order, what a full audit of a real EKS cluster found in November 2026, as published in a sample report whose cluster details are masked. The shape of each problem is what you should expect to find in a cluster nobody has audited before.
1. Databases without a backup
List every StatefulSet and PersistentVolumeClaim in production namespaces, then ask two questions of each: what makes a copy of this volume, and what is its reclaim policy. In the sample audit, all seven production databases, three MySQL, three PostgreSQL and one Neo4j, ran as a single replica with their data on one EBS volume each, every volume had reclaimPolicy Delete, and nothing in the account or the cluster backed any of them up. The only snapshots were of volumes deleted a month earlier.
Delete is the default for dynamically provisioned volumes, and it means one namespace deletion or one Argo CD prune removes the volume at once. The audit's reference case is InfluxData's 2023 postmortem: automation produced a wrong Argo CD Application name and a whole production environment was deleted. Their volumes were set to Retain and their data was backed up, so they recovered in a little under six hours without data loss. The question for your cluster is whether you would.
What to do first: take a snapshot of every production volume today, set the reclaim policy of the live volumes to Retain, and then put a snapshot policy or a dump CronJob in place. Taking the snapshot first matters because every later change in the week, including the version upgrades at the end of this list, is easier to undo when there is a copy.
2. Where the production pods actually run
Two replicas exist so one can die while the other carries on. On the same node, both go with that node, and on a spot node one reclaim is enough. List the pods of each production workload with their node and count how many share one. In the sample audit, one spot node held 20 of the 42 production pods from 15 workloads, including all three replicas of one service and both replicas of three others, plus the only pod of six single-replica workloads. Nothing declared a spread, so the scheduler kept filling the node with the most room, and a do-not-disrupt annotation on one pod kept the autoscaler from ever emptying it.
The public case behind this check is Moonlight's 2019 postmortem. Kubernetes sometimes put all three of their website pods on one node, sustained full CPU caused kernel panics, the pods moved and repeated the pattern, and the service went down on and off for four days until anti-affinity rules were added.
What to check: topologySpreadConstraints or pod anti-affinity on every production Deployment with more than one replica, a PodDisruptionBudget on each, readiness probes, and whether any pod carries an annotation that blocks node replacement. Also ask whether losing the single largest node would leave pods unschedulable; the audit calls that the N+1 check.
3. Open listeners and health checks that check nothing
Every Service of type LoadBalancer and every Ingress deserves one question: who can reach it, and should they. In the sample audit, a Kafka broker's external listener had created an internet-facing load balancer whose security group opened two ports to every address, in front of a single broker, and the load balancer carried close to three gigabytes a day, so it was in real use. The Helm chart's external access setting had created it that way without limiting source addresses.
When a database or message queue port sits on an internet-facing load balancer, one password is all that guards the data. The fix is a source range on the Service, an internal scheme, or a VPN, and the rollback is a revert of the same commit. While you are there, check the cluster's own API endpoint: an EKS public endpoint open to the whole world is a separate finding, and the audit attaches Aqua Nautilus's 2023 research on backdooring clusters through RBAC to it.
A load balancer stops sending traffic to a target its health check calls unhealthy. When every target is unhealthy, it sends traffic to all of them, so nothing looks broken and the check filters out nothing. In the sample audit, every target in four production target groups was unhealthy because the check went to the root path, where two backends answered 404 and two frontends answered a redirect. The services worked, and a dead pod would not have been removed either.
On AWS the cause is usually a missing healthcheck-path annotation on the Ingress, and the fix is one line per Ingress pointing at the same path the readiness probe uses. This check is separate from Kubernetes readiness: a pod can be Ready and still be dropped by the load balancer, or, as here, be failing and still receive traffic.
4. Versions past their support date
Read three dates: the Kubernetes minor version's end of standard support, the version range of the node autoscaler, and the age and family of the node image. In the sample audit, EKS 1.32 had left standard support eight months earlier, so the cluster was paying for extended support; Karpenter was at a version that supported Kubernetes only up to 1.30; and the node class still used Amazon Linux 2, which has no images from Kubernetes 1.33 on. Any one of the three can stop new nodes from coming up at the next upgrade.
The case the audit cites is ZenML's 2025 incident: during a production EKS upgrade, Terraform turned on KMS secret encryption, which dropped IAM permissions from the worker nodes, and pods were unreachable from outside for about eight hours. An old Kubernetes version and AMI type were among the contributing factors. Doing every overdue upgrade at once stacks up the same kind of risk, which is why the snapshots in step 1 come first and the upgrades come last.
What the first week should leave behind
By the end of the week you should have a written list of what the cluster could not show you: nodes you could not reach, logs that are not kept, audit logging that is off, dumps that may or may not run on someone's laptop. A finding you could not check is not a pass, and writing it down is what turns the first audit into a baseline the next one can be compared with.
kubegap does this audit as an Agent Skill: typed as /kubegap in Claude Code or $kubegap in Codex, it reads the cluster with the access you already have, judges 411 checks in 12 areas, and writes a report that opens with today's problems and ends with what the run could not see. The five problems above are its EKS sample report. The public incidents it cites for each check, InfluxData, Moonlight and ZenML among them, are searchable by symptom at Incident Library, https://benchlet.com/incidentlib/app, with the original write-up linked from every case.
Frequently asked questions
Should I run a full audit or start with the daily one?
Taking over a cluster is the case for a full audit: it reads inside the nodes and on the cloud side, reports five problems and writes one line per area. After that the daily audit is the routine, with three problems and a comparison with the previous daily run.
What if I only have read access?
That is enough for everything above. Listing volumes, pods, Services, Ingresses and versions needs get and list, and reading load balancer health needs Describe calls on the cloud side. An audit identity can exclude Secrets entirely; what it cannot read should be reported as not seen, not as a pass.
Why snapshots before anything else?
Because every other change in the week, from spreading pods to upgrading Kubernetes, is easier to undo when a copy exists, and because a volume with reclaimPolicy Delete can disappear with one namespace deletion. InfluxData recovered from exactly that in 2023 only because their volumes were Retain and backed up.
How do I find pods that share a node quickly?
List pods across all namespaces with the wide output, which adds the node column, and sort by node. Then, for each production workload with more than one replica, check whether a topologySpreadConstraint or anti-affinity rule exists. A node holding every replica of a service is the finding.