Incident Library
Applied filters Root causeConfiguration changeComponentCNI pluginDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
    Symptoms

    On 7 July 2023 from 11:05 to 15:25 UTC many public Jenkins infrastructure services were completely down, including get.jenkins.io, plugins.jenkins.io, accounts.jenkins.io, javadoc.jenkins.io and incrementals.jenkins.io, while ci.jenkins.io, issues.jenkins.io, repo.jenkins-ci.org and www.jenkins.io were partly or completely disrupted.

    Network unreachableResources deletedCluster or node provisioning stallService outage
    Trigger

    After the upgrade to Kubernetes 1.25 changed the range of the pods' originating IP and LDAP access was blocked, a fix (Azure PR 431) deployed at 10:55 UTC to set a proper set of pod IP ranges removed all node pools of the cluster.

    Root cause
    Configuration change
    Component
    Node poolCNI pluginCloud load balancer
    Contributing factor
    Unverified assumptionNo staged rolloutNo safeguard on destructive change
    Detection

    After the upgrade, at 10:30 UTC, the team noticed that services inside the cluster, such as accounts.jenkins.io, could not reach the LDAP service.

    Remediation
    • Created the cluster again from scratch with a fixed network setup
    • Set a custom pod CIDR in a range separate from the virtual network
    • Moved the services and their DNS records to new public IPs
    • Pushed a pull request that recorded the manual rebuild, including the IP setup, in code
    Action items
    • Protect the public IPs from deletion with a Management Lock
    • Keep the public IPs in a separate resource group and point the Kubernetes-managed load balancers to it with the service.beta.kubernetes.io/azure-load-balancer-resource-group annotation
    • Improve network diagrams and documentation so possible overlaps are visible when preparing operations
    • Avoid changing the configuration of all AKS node pools at once
    Lessons

    The team said that changing one node pool at a time would have shown the problem after the first pool and avoided a full outage, and that the public IPs should change as little as possible because users behind corporate firewalls with allow-lists depend on them. The team also noted that it could rebuild, in under 4 hours, the cluster that hosts almost all public services.

    View case
  2. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
    Symptoms

    At 4:53 pm Pacific on 25 January 2023 one of OpenAI's GPU clusters had a major outage, and the text-davinci-003 model went fully down.

    Service outageThroughput degradation
    Trigger

    A configuration change to the CNI (Container Network Interface) plugins, which provide network connectivity to containers, was deployed to a GPU cluster that contained a hardware variant the staging environment did not have.

    Root cause
    Configuration change
    Component
    CNI plugin
    Contributing factor
    Unrepresentative stagingMonitoring blind spot
    Detection

    Engineers immediately identified the problem as network connectivity, but it took nearly an hour to identify CNI as the cause.

    Remediation
    • put a fix for the CNI configuration change in place, which restored 80% of traffic immediately
    • deployed models back onto GPUs over about three more hours until capacity was fully stable
    Action items
    • Make staging environments contain every type of hardware configuration so they better represent production
    • Monitor and alert on the health of the CNI system directly
    • Fix the network and storage bottlenecks that limit how quickly many replicas of a model can be deployed
    Lessons

    A change that had been deployed to other clusters over 24 hours and deemed safe still broke a cluster with different hardware, so staging needs every hardware configuration that production has.

    View case
Check your cluster for this

These 2 cases involve CNI plugin and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
CNI plugin2Cloud load balancer1Node pool1
Contributing factor
Unverified assumption1Monitoring blind spot1No safeguard on destructive change1No staged rollout1Unrepresentative staging1
Organisation
Jenkins project1OpenAI1
  1. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
  2. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023