Incident Library
Applied filters ComponentIngress controllerSymptomStale endpoint routingOrganisationCrestaClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. Symptoms

    During a deployment, traffic into the Kubernetes cluster started hitting dead ends: some API requests timed out and requests were slow and disrupted while the rollout ran.

    Trigger

    A deployment rollout changed backend endpoints while a long-running NGINX full reload was in progress, with the enable-serial-reloads option enabled on the ingress-nginx controller.

    Root cause
    Component
    Contributing factor
    –
    Remediation
    • restarted the NGINX Ingress controller as an attempt to resolve the problem
    Action items
    • Open an issue in kubernetes/ingress-nginx describing how enable-serial-reloads blocks dynamic endpoint updates
    Lessons

    A flag meant only to serialise full reloads also delayed endpoint updates, which opened a window during rollouts in which the ingress routed to pods that no longer existed. Options that change how the ingress controller applies configuration need to be checked against what happens to endpoint updates during a long reload.

Check your cluster for this

This case involves Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

By root cause1