Cases21 cases

14 to 18 of 21

1. IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its type

   Hyperconnect2023

   Symptoms

   Hyperconnect upgraded the Fluent Bit sidecar of one service with little user impact to Fluent Bit 2.0 and watched it for several days. Only the containers that used the Kafka output plugin were being restarted, each time on SIGSEGV. The stack trace printed at the crash pointed into the Kafka output plugin, but no core file was left behind, because Fluent Bit ran inside Kubernetes.

   Container crash loop

   Trigger

   Upgrading the Fluent Bit logging sidecar to version 2.0 (the 2.0.3 image) on a service with low user impact.

   Root cause

   Software defect

   Component

   Telemetry agent

   Contributing factor

   Limited debug access

   Detection

   The team monitored the upgraded sidecars for several days after the upgrade and saw that the containers using the Kafka output plugin were restarting with SIGSEGV.

   Remediation

   • patched the Fluent Bit Kafka output plugin to skip data that is not a map before calling produce_message()
   • ran a custom build with the patch as the sidecar, which stopped the segmentation faults
   • submitted the patch upstream as fluent-bit pull request 6452, which was merged eight days later

   Lessons

   A segmentation fault in a container can be debugged from its core file, but Kubernetes does not keep one by default: a volume mounted into the pod and a core_pattern pointing into it on the node are needed first. The debug image of the same version, run locally on the same CPU architecture with the matching source checkout, gives a gdb environment that matches production. Core files contain part of the process memory and may hold sensitive data, so they should be shared with care.

   View case
2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes

   DFDS2022

   Symptoms

   At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

   Deploys failingContainer crash loop

   Trigger

   A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

   Root cause

   Certificate or credential expiry

   Component

   Policy engineAdmission webhook

   Contributing factor

   Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal

   Detection

   An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

   Remediation

   • reproduced the issue in a sandbox cluster and deployed the fix there first
   • updated Kyverno to a newer Helm chart version in the affected production cluster

   Action items

   • Implement monitoring on Kyverno
   • Create internal documentation on the Kyverno setup
   • Investigate the exact certificate issue in the third-party application
   • Consider changing the failurePolicy on ClusterPolicy resources going forward

   Lessons

   Operational issues should be added to the 'Ops Day to Day' board from the start.

   View case
3. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage

   GoCardless2020

   Symptoms

   On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

   Service outageContainer crash loopTLS certificate errorsCredentials unavailable

   Trigger

   At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

   Root cause

   Certificate or credential expiry

   Component

   cert-managerSecrets store

   Contributing factor

   Single point of failureMonitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

   Remediation

   • Forced cert-manager to recreate the Vault certificate
   • Restarted the Vault cluster by hand so it loaded the new certificate
   • Deleted some pods of critical workloads to skip their 5-minute exponential backoff

   Action items

   • Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
   • Deploy the Reloader controller to roll Vault when its certificate data changes
   • Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup

   Lessons

   A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

   View case
4. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory

   UK Ministry of Justice2020

   Symptoms

   At 10:43 on 14 August 2020 two of the six ingress controller pods in the cluster were in a crash loop. Restarting those pods did not fix them.

   Container crash loop

   Root cause

   Capacity exhaustion

   Component

   Ingress controller

   Contributing factor

   Harmful manual actionSingle point of failure

   Detection

   Two of the six ingress controller pods were seen crash-looping at 10:43; the incident was declared at 11:01.

   Remediation

   • restarted the leader ingress controller pod so the other replicas could connect and get the latest nginx configuration

   View case
5. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999

   LINE2020

   Symptoms

   One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.

   Container crash loopOOMKilledMonitoring outageProbe failures

   Trigger

   After start, the Prometheus time series count rose to 2,162,256 within five minutes and its memory use filled the 8GB host.

   Root cause

   Capacity exhaustion

   Component

   PrometheuskubeletLinux kernel

   Contributing factor

   No resource requests or limitsNo capacity planning

   Detection

   The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.

   Remediation

   • removed most of the unnecessary settings from the Prometheus ConfigMap, after which the container stopped crashing

   Action items

   • Set appropriate resource requests and limits on the Prometheus pod
   • Reduce unneeded metrics by adjusting scrape targets or scrape interval
   • Estimate Prometheus memory needs, provide dedicated hosts with a node label such as deploy-topology: Prometheus, and pin the Prometheus pod to them with a nodeSelector

   Lessons

   Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.

   View case

By root cause8