Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them

Symptoms

On 7 July 2023 from 11:05 to 15:25 UTC many public Jenkins infrastructure services were completely down, including get.jenkins.io, plugins.jenkins.io, accounts.jenkins.io, javadoc.jenkins.io and incrementals.jenkins.io, while ci.jenkins.io, issues.jenkins.io, repo.jenkins-ci.org and www.jenkins.io were partly or completely disrupted.

It started at 10:30 UTC, after a successful upgrade of the public Kubernetes cluster in Azure to 1.25: services inside the cluster, such as accounts.jenkins.io, could not reach the LDAP service. The pods' originating IP was now in a different range than before, and IP restrictions blocked the requests.

The fix deployed at 10:55 UTC to set a proper set of pod IP ranges removed all the node pools and failed to re-create them, so every service in the cluster went down. The re-creation failed because the cluster's virtual network, 10.244.0.0/16, overlapped 10.244.0.0/24, the default pod CIDR in Azure when kubenet is used.

When the original cluster was deleted, its public IP addresses were removed with it, so the services came back on new addresses, 20.7.178.24 and 2603:1030:408:5::15a, and users behind corporate networks had to update their DNS, proxy and firewall settings.

Symptom Network unreachableResources deletedCluster or node provisioning stallService outage

Root cause

Trigger

After the upgrade to Kubernetes 1.25 changed the range of the pods' originating IP and LDAP access was blocked, a fix (Azure PR 431) deployed at 10:55 UTC to set a proper set of pod IP ranges removed all node pools of the cluster.

Detection

After the upgrade, at 10:30 UTC, the team noticed that services inside the cluster, such as accounts.jenkins.io, could not reach the LDAP service.

Remediation

• Created the cluster again from scratch with a fixed network setup
• Set a custom pod CIDR in a range separate from the virtual network
• Moved the services and their DNS records to new public IPs
• Pushed a pull request that recorded the manual rebuild, including the IP setup, in code

Action items

• Protect the public IPs from deletion with a Management Lock
• Keep the public IPs in a separate resource group and point the Kubernetes-managed load balancers to it with the service.beta.kubernetes.io/azure-load-balancer-resource-group annotation
• Improve network diagrams and documentation so possible overlaps are visible when preparing operations
• Avoid changing the configuration of all AKS node pools at once

Lessons

The team said that changing one node pool at a time would have shown the problem after the first pool and avoided a full outage, and that the public IPs should change as little as possible because users behind corporate firewalls with allow-lists depend on them. The team also noted that it could rebuild, in under 4 hours, the cluster that hosts almost all public services.

Sources

• https://www.jenkins.io/blog/2023/07/12/jenkins-mirrors-postmortem-outage/ · blog · archived copy

Related cases

• IL-0346 Render's routing service returned intermittent 404s for web services and static sites in OregonRender · 2025

  same root cause · shared Cloud load balancer

• IL-0543 Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutesTwingate · 2025

  same root cause · shared Cloud load balancer

• IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025

  same root cause · shared Cloud load balancer

• IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024

  same root cause · shared Node pool

• IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023

  same root cause · shared Node pool

• IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023

  same root cause · shared Node pool

• IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023

  same root cause · shared Node pool

• IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut downStytch · 2023

  same root cause · shared Node pool

• IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

  same root cause · shared Node pool

• IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully downOpenAI · 2023

  same root cause · shared CNI plugin

• IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listenerPython Software Foundation · 2022

  same root cause · shared Cloud load balancer

• IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE Plus · 2022

  same root cause · shared Cloud load balancer

• IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022

  same root cause · shared Cloud load balancer

• IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021

  same root cause · shared Cloud load balancer

• IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021

  same root cause · shared Cloud load balancer

• IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster serviceDFDS · 2020

  same root cause · shared Cloud load balancer

• IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019

  same root cause · shared Cloud load balancer

• IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018

  same root cause · shared Node pool

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0473
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0473/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.