IL-0521 Jenkins project · 2021-06-02kubernetesazure

AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired

AKS nodes use the cluster's Azure Service Principal credential to authenticate to the Azure API, which gives them the rights to mount file system volumes such as Azure Blob or CIFS, to use private network access and to create and delete Azure load balancers. The Jenkins infrastructure cluster's credential expired at 17:53 UTC on 2 June 2021. From 18:02 UTC pods could not mount volumes and scaled-up nodes could not get a network virtual interface from the CNI, so pods with persistent volumes could not be scheduled and builds using them failed on trusted.ci.jenkins.io, infra.ci.jenkins.io and release.ci.jenkins.io, including the LTS release process.

The error was found in the AKS UI console during the investigation of the LTS 2.289.1 release problem. At 15:15 UTC on 3 June the infrastructure team renewed the service principal credential. Resetting it requires a full restart of the nodes, which AKS carried out automatically with the standard cordon, drain, update and reboot process, and the operation could not be stopped. The restart made ldap.jenkins.io unavailable for two minutes and plugins.jenkins.io answer HTTP/500 on 33% of requests. By 16:35 UTC every operation had finished, all Kubernetes-hosted applications had restarted and no more errors showed in logs or metrics. One engineer did not have the full Azure authorisation needed to reset the service principal, so another team member did the reset.

Severity: –
Scope: Pods with persistent volumes could not be scheduled from 18:02 UTC on 2 June to 15:30 UTC on 3 June, failing builds on trusted.ci.jenkins.io, infra.ci.jenkins.io and release.ci.jenkins.io; during the node restarts ldap.jenkins.io was down for two minutes and plugins.jenkins.io returned HTTP/500 on 33% of requests
Users affected: –
Region: –
Time to detect: –
Time to resolve: 22h 42m

Symptoms

From 18:02 UTC on 2 June 2021, the Jenkins infrastructure AKS cluster logged errors about pods that were unable to mount volumes and about scaled-up nodes that were unable to obtain a network virtual interface from the CNI.

Pods with a Persistent Volume could not be scheduled, so builds that used a pod with a PVC failed on trusted.ci.jenkins.io, infra.ci.jenkins.io and release.ci.jenkins.io. The errors also hit the LTS release process for 2.289.1, and the cause was found in the AKS UI console while the team investigated that release problem: the cluster's Azure Service Principal credential had expired at 17:53 UTC.

Renewing the credential at 15:15 UTC on 3 June started an automatic operation on the AKS cluster that could not be stopped and restarted all nodes. During the restart ldap.jenkins.io was unavailable from 15:17 to 15:19 UTC, and plugins.jenkins.io answered HTTP/500 on 33% of requests until 16:35 UTC.

Symptom Credentials unavailable Pods stuck in ContainerCreating

Root cause

Root cause Certificate or credential expiry

Component Cluster cloud identity Persistent volumes and stateful sets CNI plugin Node pool

Contributing factor Monitoring blind spot

Trigger: The Azure Service Principal credential of the AKS cluster expired at 17:53 UTC on 2 June 2021.

Detection

The error behind the failures was found in the AKS UI console while the team investigated the root cause of the LTS 2.289.1 release problem.

Remediation

Renewed the AKS cluster's Azure Service Principal credential, which made AKS restart all nodes automatically

Action items

Add an automatic PagerDuty check that alerts a week before the service principal expires, as is done for certificates
Keep a team-level calendar for manual operations such as resetting a credential
Consider managed identities, which renew automatically, instead of the service principal

Lessons

–

Sources

https://github.com/jenkins-infra/documentation/blob/main/postmortem/20210602-AKS-incident.md · github

References

Related cases

IL-0158 An expired credential caused authentication failures that stopped LKE and LKE-E from deploying clusters and nodes in all regions, which also blocked node recycling and cluster autoscalingLinode · 2026
same root cause · shared Node pool
IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effectWebhook Relay · 2022
same root cause · shared Node pool

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cluster cloud identity, Persistent volumes and stateful sets, CNI plugin among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0521
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0521/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.