Incident Library

Organisation

Jenkins project

Jenkins project · jenkins-project

3 cases · 2021 to 2023

Root cause Certificate or credential expiry1Configuration change1Breaking change in an upgrade1
Component CNI plugin2Node pool2Admission webhook1CDN and edge network1Cloud load balancer1Cluster cloud identity1Helm1Ingress controller1Persistent volumes and stateful sets1
Contributing factor Unverified assumption2Monitoring blind spot2Cache-masked failure1Harmful manual action1No safeguard on destructive change1No staged rollout1
Symptom Resources deleted2Service outage2Cluster or node provisioning stall1Credentials unavailable1Network unreachable1Pods stuck in ContainerCreating1

Cases

  1. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate themJenkins project · 2023

    Root cause Configuration change

  2. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hoursJenkins project · 2021

    Root cause Breaking change in an upgrade

  3. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expiredJenkins project · 2021

    Root cause Certificate or credential expiry