Cases52 cases

26 to 30 of 52

1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes

   DFDS2022

   Symptoms

   At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

   Deploys failingContainer crash loop

   Trigger

   A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

   Root cause

   Certificate or credential expiry

   Component

   Policy engineAdmission webhook

   Contributing factor

   Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal

   Detection

   An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

   Remediation

   • reproduced the issue in a sandbox cluster and deployed the fix there first
   • updated Kyverno to a newer Helm chart version in the affected production cluster

   Action items

   • Implement monitoring on Kyverno
   • Create internal documentation on the Kyverno setup
   • Investigate the exact certificate issue in the third-party application
   • Consider changing the failurePolicy on ClusterPolicy resources going forward

   Lessons

   Operational issues should be added to the 'Ops Day to Day' board from the start.

   View case
3. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment

   InfluxData2022

   Symptoms

   Right after a pull request to the infrastructure-as-code repository was merged, every API endpoint of one InfluxDB Cloud production cluster began returning 404 errors. Writes and reads failed alike: no one could collect data, scheduled tasks failed to run and external queries did not work.

   Service outageResources deletedHigh error rate

   Trigger

   A merged pull request whose rendered YAML added a new Argo CD Application and AppProject that an automation error had named app01-cluster01, the name of the existing production Application, instead of app02-cluster01; reviewers did not notice the app01 and app02 difference.

   Root cause

   Configuration change

   Component

   Argo CDPersistent volumes and stateful setsConfiguration templating

   Contributing factor

   Missed in reviewNo runbookNo safeguard on destructive changeMonitoring blind spotAutomation amplification

   Detection

   Monitoring and alerting did not detect the issue right away, so the team's first response was to try to roll the change back rather than plan a recovery. The write-up does not say how the team found out.

   Remediation

   • stopped the revert of the merged pull request
   • restored the etcd metadata cluster
   • recreated the missing Kubernetes objects so the retained PersistentVolumes became available, then recreated the StatefulSets
   • restored the Zookeeper volumes from an hourly Velero backup
   • rebuilt the Kafka StatefulSet one pod at a time with health and readiness checks disabled
   • accepted writes into the Kafka write-ahead log and scaled up the write path with more replicas and higher memory requests and limits
   • recreated the storage pods, which downloaded data from object storage, reindexed it and replayed the write-ahead log
   • re-enabled Argo CD, queries, dashboards and scheduled tasks, restored replica counts and ran final QA checks

   Action items

   • Write runbooks for restoring state when Kubernetes objects such as PersistentVolumeClaims are deleted but the data on the underlying volumes is preserved
   • Set every volume in every environment to retain data when its PersistentVolumeClaim is deleted
   • Improve the process for handling public-facing incidents
   • Store generated YAML as one object per file, so an overwrite of an existing object shows up as such in review
   • Detect duplicate objects when generating YAML and warn before review, treating objects with the same group, namespace and name as the same even when the apiVersion string differs
   • Add the Argo CD Prune=false annotation to all stateful resources and to Namespace objects managed by Argo CD
   • Set FailOnSharedResource=true on Argo CD Applications so Argo CD fails before changing an object another Application manages or managed

   Lessons

   The biggest gap was that monitoring did not notice the deletion quickly, which pushed the team toward a reflex rollback; in a stateful system a revert can create a new, empty instance instead of bringing the old one back. Retained volumes and clients that retry writes are what kept data loss at zero. Rendering many objects into one file let an overwrite of an existing object look like an addition, so generated configuration needs one object per file and duplicate detection that compares objects the way Kubernetes does. Stateful resources need protection from pruning, and that protection must cover their Namespace too, or deleting the namespace removes everything inside it.

   View case
4. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use

   Binder Project2022

   Symptoms

   On 2 June 2022 from 10:00 CET, mybinder.org stopped launching any new pods, and nobody on the team noticed until a user reported a Binder outage in the Matrix channel at 21:00.

   Deploys failingService outage

   Root cause

   Software defect

   Component

   GKE control plane

   Contributing factor

   Monitoring blind spot

   Detection

   A user reported a Binder outage in the Matrix channel at 21:00, and a team member who saw it found that pods had not been launching for several hours.

   Remediation

   • Deleted the gke-resource-quotas ResourceQuota, after which pods could be created again

   Action items

   • Address uptime monitoring and alerting

   Lessons

   It took 11 hours to realise there was a major outage. Once the right object was known, deleting it fixed the problem quickly, and the team was lucky that someone with the skills and permissions to make the change was at their computer at 11pm their time.

   View case
5. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case

By root cause9