Incident Library
Applied filters SymptomCluster or node provisioning stallOrganisationJenkins projectClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
    Symptoms

    On 7 July 2023 from 11:05 to 15:25 UTC many public Jenkins infrastructure services were completely down, including get.jenkins.io, plugins.jenkins.io, accounts.jenkins.io, javadoc.jenkins.io and incrementals.jenkins.io, while ci.jenkins.io, issues.jenkins.io, repo.jenkins-ci.org and www.jenkins.io were partly or completely disrupted.

    Network unreachableResources deletedCluster or node provisioning stallService outage
    Trigger

    After the upgrade to Kubernetes 1.25 changed the range of the pods' originating IP and LDAP access was blocked, a fix (Azure PR 431) deployed at 10:55 UTC to set a proper set of pod IP ranges removed all node pools of the cluster.

    Root cause
    Configuration change
    Component
    Node poolCNI pluginCloud load balancer
    Contributing factor
    Unverified assumptionNo staged rolloutNo safeguard on destructive change
    Detection

    After the upgrade, at 10:30 UTC, the team noticed that services inside the cluster, such as accounts.jenkins.io, could not reach the LDAP service.

    Remediation
    • Created the cluster again from scratch with a fixed network setup
    • Set a custom pod CIDR in a range separate from the virtual network
    • Moved the services and their DNS records to new public IPs
    • Pushed a pull request that recorded the manual rebuild, including the IP setup, in code
    Action items
    • Protect the public IPs from deletion with a Management Lock
    • Keep the public IPs in a separate resource group and point the Kubernetes-managed load balancers to it with the service.beta.kubernetes.io/azure-load-balancer-resource-group annotation
    • Improve network diagrams and documentation so possible overlaps are visible when preparing operations
    • Avoid changing the configuration of all AKS node pools at once
    Lessons

    The team said that changing one node pool at a time would have shown the problem after the first pool and avoided a full outage, and that the public IPs should change as little as possible because users behind corporate firewalls with allow-lists depend on them. The team also noted that it could rebuild, in under 4 hours, the cluster that hosts almost all public services.

    View case
Check your cluster for this

This case involves Cloud load balancer and CNI plugin. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Cloud load balancerCNI pluginNode pool
Contributing factor
Unverified assumptionNo safeguard on destructive changeNo staged rollout
Organisation
Jenkins project
  1. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023