Cases3 cases

1 to 3 of 3

1. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them

   Jenkins project2023

   Symptoms

   On 7 July 2023 from 11:05 to 15:25 UTC many public Jenkins infrastructure services were completely down, including get.jenkins.io, plugins.jenkins.io, accounts.jenkins.io, javadoc.jenkins.io and incrementals.jenkins.io, while ci.jenkins.io, issues.jenkins.io, repo.jenkins-ci.org and www.jenkins.io were partly or completely disrupted.

   Network unreachableResources deletedCluster or node provisioning stallService outage

   Trigger

   After the upgrade to Kubernetes 1.25 changed the range of the pods' originating IP and LDAP access was blocked, a fix (Azure PR 431) deployed at 10:55 UTC to set a proper set of pod IP ranges removed all node pools of the cluster.

   Root cause

   Configuration change

   Component

   Node poolCNI pluginCloud load balancer

   Contributing factor

   Unverified assumptionNo staged rolloutNo safeguard on destructive change

   Detection

   After the upgrade, at 10:30 UTC, the team noticed that services inside the cluster, such as accounts.jenkins.io, could not reach the LDAP service.

   Remediation

   • Created the cluster again from scratch with a fixed network setup
   • Set a custom pod CIDR in a range separate from the virtual network
   • Moved the services and their DNS records to new public IPs
   • Pushed a pull request that recorded the manual rebuild, including the IP setup, in code

   Action items

   • Protect the public IPs from deletion with a Management Lock
   • Keep the public IPs in a separate resource group and point the Kubernetes-managed load balancers to it with the service.beta.kubernetes.io/azure-load-balancer-resource-group annotation
   • Improve network diagrams and documentation so possible overlaps are visible when preparing operations
   • Avoid changing the configuration of all AKS node pools at once

   Lessons

   The team said that changing one node pool at a time would have shown the problem after the first pool and avoided a full outage, and that the public IPs should change as little as possible because users behind corporate firewalls with allow-lists depend on them. The team also noted that it could rebuild, in under 4 hours, the cluster that hosts almost all public services.

   View case
2. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down

   OpenAI2023

   Symptoms

   At 4:53 pm Pacific on 25 January 2023 one of OpenAI's GPU clusters had a major outage, and the text-davinci-003 model went fully down.

   Service outageThroughput degradation

   Trigger

   A configuration change to the CNI (Container Network Interface) plugins, which provide network connectivity to containers, was deployed to a GPU cluster that contained a hardware variant the staging environment did not have.

   Root cause

   Configuration change

   Component

   CNI plugin

   Contributing factor

   Unrepresentative stagingMonitoring blind spot

   Detection

   Engineers immediately identified the problem as network connectivity, but it took nearly an hour to identify CNI as the cause.

   Remediation

   • put a fix for the CNI configuration change in place, which restored 80% of traffic immediately
   • deployed models back onto GPUs over about three more hours until capacity was fully stable

   Action items

   • Make staging environments contain every type of hardware configuration so they better represent production
   • Monitor and alert on the health of the CNI system directly
   • Fix the network and storage bottlenecks that limit how quickly many replicas of a model can be deployed

   Lessons

   A change that had been deployed to other clusters over 24 hours and deemed safe still broke a cluster with different hardware, so staging needs every hardware configuration that production has.

   View case
3. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3

   Tinder2019

   Symptoms

   In the early morning of 8 January 2019 Tinder's platform had a persistent outage. Communication from nodes to pods failed, and so did DNS lookups, because DNS ran inside the Kubernetes cluster.

   Service outageNetwork unreachableDNS resolution failures

   Trigger

   Earlier that morning, pod and node counts on the cluster were scaled up in response to an unrelated increase in platform latency.

   Root cause

   Capacity exhaustion

   Component

   Linux kernelCNI plugin

   Contributing factor

   Unreviewed defaults

   Remediation

   • Raised gc_thresh1, gc_thresh2 and gc_thresh3 on the nodes
   • Restarted Flannel so it re-registered the missing networks

   Lessons

   Tinder lists the outage among its learnings about network fabric limits: with Flannel over VXLAN, every node, and every pair of nodes that exchange pod traffic, adds entries to each node's ARP table, so at 605 nodes the default gc_thresh3 hard cap was exceeded on all nodes.

   View case

By root cause2