Cases4 cases

1 to 4 of 4

1. IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was running

   Cresta2025

   Symptoms

   During a deployment, traffic into the Kubernetes cluster started hitting dead ends: some API requests timed out and requests were slow and disrupted while the rollout ran.

   Intermittent timeoutsStale endpoint routingHigh latency

   Trigger

   A deployment rollout changed backend endpoints while a long-running NGINX full reload was in progress, with the enable-serial-reloads option enabled on the ingress-nginx controller.

   Root cause

   Software defect

   Component

   Ingress controller

   Contributing factor

   –

   Remediation

   • restarted the NGINX Ingress controller as an attempt to resolve the problem

   Action items

   • Open an issue in kubernetes/ingress-nginx describing how enable-serial-reloads blocks dynamic endpoint updates

   Lessons

   A flag meant only to serialise full reloads also delayed endpoint updates, which opened a window during rollouts in which the ingress routed to pods that no longer existed. Options that change how the ingress controller applies configuration need to be checked against what happens to endpoint updates during a long reload.

   View case
2. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB

   Lotte E-commerce2022

   Symptoms

   LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.

   High error rateIntermittent timeoutsStale endpoint routing

   Trigger

   HPA scale-in deleting pods, and the deletion of Blue pods after a blue/green switch of the Service label selector.

   Root cause

   DNS or network fault

   Component

   Horizontal pod autoscalerPod terminationkube-proxyconntrack tableCloud load balancerIngress controller

   Contributing factor

   Unreviewed defaults

   Remediation

   • delayed container shutdown on pod deletion (a wait in the SIGTERM handler or a preStop sleep, capped by terminationGracePeriodSeconds) so containers stop only after the pod has left the Service endpoints
   • switched some workloads' Ingress-created ALBs from instance mode to IP mode so traffic goes straight to pod IPs without iptables and conntrack

   Action items

   • Move all workloads to ALB IP mode gradually

   Lessons

   Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application. With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.

   View case
3. IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone

   UK Ministry of Justice2022

   Symptoms

   On 11 July 2022 users reported in the #ask-cloud-platform Slack channel that their applications on the Ministry of Justice Cloud Platform were slow some of the time. About 25% of ingress traffic was affected: after an AWS incident the day before, one of the three network interfaces of the 'default' ingress controllers performed slowly, and clients timed out when they tried to connect to one of the Elastic IPs of the Network Load Balancer (NLB) in front of them.

   High latencyIntermittent timeouts

   Trigger

   Some instances of the AWS NLB health checking subsystem were restarted, which set off a latent software race condition in that subsystem.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerIngress controller

   Contributing factor

   –

   Detection

   Users reported slow performance of their applications in the #ask-cloud-platform Slack channel.

   Remediation

   • AWS restarted the NLB health checking subsystem, which refreshed its list of targets and recovered the NLB in the affected Availability Zone

   Action items

   • Raised mitigation tickets after a post-incident review, filed as post-aws-incident issues in the ministryofjustice/cloud-platform repository

   View case
4. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded

   OpenAI2022

   Symptoms

   At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.

   Service outageHigh error rateIntermittent timeouts

   Trigger

   A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.

   Root cause

   Operator error

   Component

   Ingress controllerModel hub

   Contributing factor

   No safeguard on destructive changeNo headroomDeferred upgradesOverlapping failures

   Remediation

   • re-allocated the models from the lost cluster to the remaining two clusters
   • rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
   • built workarounds with Azure support after losing the connection to Azure Private Link
   • tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
   • moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT

   Action items

   • Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
   • Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
   • Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
   • Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
   • Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
   • Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
   • Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
   • Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
   • Change how model weights are stored so they are more efficient and faster to load

   Lessons

   Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.

   View case

By root cause4