Cases32 cases

11 to 15 of 32

1. IL-0065 Any pod could take over the cluster by rewriting Argo CD's unauthenticated Redis cache (CVE-2024-31989)

   Cycode2024

   Symptoms

   This is a vulnerability report, not an outage, so nothing failed visibly. What an engineer can check is whether the Argo CD Redis server accepts connections without a password from a pod in another namespace, on port 6379 at the address kube-dns returns. In the default installation it did.

   Unauthorised access or a foreign workload

   Trigger

   Cycode Labs installed Argo CD with the default configuration on a new Amazon EKS cluster and connected to its Redis server from a low-privilege pod in another namespace, simulating a compromised pod.

   Root cause

   Software defect

   Component

   Argo CD

   Contributing factor

   Unreviewed defaultsExcessive blast radiusCredential in configuration

   Detection

   Found by Cycode Labs security research, not by an operator.

   Remediation

   • Argo CD released 2.11.1, 2.10.10, 2.9.15 and 2.8.19 with an initialized password for the Redis server

   Lessons

   The hash that checks the cached manifests is computed without a secret key, so anyone who can write to Redis can recompute it and make Argo CD deploy an altered manifest. Cycode recommends upgrading to the fixed versions, checking that the argocd-redis-network-policy NetworkPolicy exists and that a CNI enforces it, using controller-based secret management instead of injecting secrets through Argo CD plugins so secrets never reach Redis, and running Argo CD on a dedicated cluster.

   View case
2. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast

   Microsoft Azure2023

   Symptoms

   From about 10:30 UTC on 30 August 2023, customers had trouble accessing or using Azure, Microsoft 365 and Power Platform services in Australia East. The affected Azure services included Azure Kubernetes Service (AKS) along with storage, SQL Database, Cosmos DB, virtual machines and Azure Resource Manager.

   Control plane unresponsiveHigh error rateService outage

   Trigger

   At 08:41 UTC a lightning strike on electrical infrastructure about 18 miles away caused a utility voltage sag that shut down the cooling chillers of datacenters in one Australia East availability zone; 13 chillers failed to restart automatically and five could not be restarted, so two data halls overheated and their infrastructure was shut down.

   Root cause

   Cloud provider outage

   Component

   AKS control planeApplication database

   Contributing factor

   Single point of failureExcessive blast radius

   Detection

   Cosmos DB impact was detected by monitoring at 10:47 UTC, first automated communications went to Azure Service Health at 10:48 UTC, the public Azure Status page was updated at 11:30 UTC, and all subscriptions in Australia East were sent portal communications at 11:36 UTC.

   Remediation

   • shut down the remaining networking, compute and storage in the two affected data halls so the chilled water loop could cool down
   • restarted the five chillers manually at 12:12 UTC
   • restored power to all hardware by 15:10 UTC and brought infrastructure back in phases
   • restored AKS operations through the recovery of the SQL Database operation queue, with no other mitigation

   Action items

   • AKS: convert all operation queue SQL Database databases to be zone redundant (estimated September 2023)
   • AKS: replace all cross-region SQL Database queue usage with zone-redundant Service Bus queues (estimated September 2023)
   • Change the control timing logic of the Chiller Management System so chillers do not fail to restart after voltage fluctuations
   • Change the emergency procedure for manual restarts of simultaneous chiller failures from an adjacent sequence to a load-based sequence per data hall
   • Increase technician staffing at the datacenter as a temporary measure for manual chiller restarts
   • Storage: optimise the large-scale recovery process and update troubleshooting guides (estimated December 2023)
   • Azure Service Fabric: improve reliability of SQL Database tenant ring recovery (estimated December 2023)
   • SQL Database: review the trigger criteria of auto-failover groups (estimated October 2023)
   • SQL Database: upgrade internal tooling to enable mass migration of databases (estimated December 2023)
   • Cosmos DB: optimise Service Managed Failover for single-region-write accounts (estimated November 2023)
   • ARM: complete the migration of its storage layer to the zonally redundant architecture (estimated December 2023)
   • Incident management: harden readiness, process and playbook for power-down scenarios (estimated October 2023)

   Lessons

   A control plane spread across availability zones still fails with a zone when one of its dependencies, here the SQL Database operation queue, is not zone redundant. When a second region uses the same database, the failure crosses into that region as well. Workloads already running were not expected to be affected; what stopped was everything that needs the managed service's control plane, such as scaling and cluster changes.

   View case
3. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together

   UK Ministry of Justice2023

   Symptoms

   At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.

   Service outage

   Trigger

   The team changed the instance type of the cluster's worker nodes through Terraform.

   Root cause

   Configuration change

   Component

   Node poolTerraform

   Contributing factor

   Excessive blast radiusNo runbook

   Detection

   Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.

   Action items

   • Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
   • Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type

   View case
4. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up

   2i2c2023

   Symptoms

   The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

   Pod evictionPods Pending

   Trigger

   A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode pool

   Contributing factor

   No headroomExcessive blast radiusMonitoring blind spot

   Detection

   A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

   Action items

   • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
   • Make the Grafana OOMKiller dashboard work (#2213)
   • Move LEAP cloud infrastructure to shared nodes (#2209)
   • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
   • Decide whether Prometheus should scrape dask worker pods (#2366)

   Lessons

   Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

   View case
5. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx

   LINE+2022

   Symptoms

   The graph of WebSocket connections to messaging-hub, the messaging platform used by the Japanese food delivery service Demaecan, showed sudden deep drops even outside peak traffic. The drops kept coming back at irregular intervals. Messages still reached clients, because messaging-hub switches to Firebase Cloud Messaging when a WebSocket is gone.

   Connection resetsDatabase overloadIntermittent connection failuresHigh latency

   Trigger

   Another service that shared the same L7 load balancer deployed its own load balancer configuration in the Verda dashboard, which reloaded HAProxy; 30 minutes later, at the hard-stop-after limit, the old HAProxy process exited and closed every WebSocket connection it still held.

   Root cause

   Configuration change

   Component

   HAProxyCloud load balancerconntrack tableIngress controller

   Contributing factor

   Change with a delayed effectExcessive blast radiusRetry stormUnverified assumption

   Detection

   The developer noticed deep, sudden drops in the WebSocket connection graph that were not tied to peak traffic and kept recurring irregularly. The Nginx packet drops were found through network monitoring.

   Remediation

   • separated the Merchant system's token issuance API into its own deployment so reconnect traffic did not delay other Merchant APIs
   • scaled out Nginx with pod anti-affinity and a label selector so each pod ran on a different node, which stopped the nf_conntrack packet drops
   • raised Ingress Nginx proxy-read-timeout and proxy-send-timeout and the load balancer timeout to 3600s, which did not stop the drops
   • moved messaging-hub from the shared L7 load balancer to an L4 load balancer, which ended the WebSocket drops

   Lessons

   A reload of a shared load balancer by a different team can cut long-lived connections, and with hard-stop-after the cut comes 30 minutes later, far from the change that caused it. The author's lesson is not to narrow the scope of the search too early: trusting that the infrastructure was fine kept the investigation inside the team's own systems for days. A reconnect wave from many WebSocket clients can overload the systems that issue connection tokens and fill nf_conntrack on proxy nodes. Scaling Nginx out only helps for conntrack if the pods land on different nodes, and anti-affinity leaves extra replicas Pending once they outnumber the nodes. That messaging-hub problems can still load the connected systems remains an open problem.

   View case

By root cause8