Incident Library
Applied filters Root causeDNS or network faultComponentconntrack tableSymptomHigh error rateClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
    Symptoms

    LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.

    High error rateIntermittent timeoutsStale endpoint routing
    Trigger

    HPA scale-in deleting pods, and the deletion of Blue pods after a blue/green switch of the Service label selector.

    Root cause
    DNS or network fault
    Component
    Horizontal pod autoscalerPod terminationkube-proxyconntrack tableCloud load balancerIngress controller
    Contributing factor
    Unreviewed defaults
    Remediation
    • delayed container shutdown on pod deletion (a wait in the SIGTERM handler or a preStop sleep, capped by terminationGracePeriodSeconds) so containers stop only after the pod has left the Service endpoints
    • switched some workloads' Ingress-created ALBs from instance mode to IP mode so traffic goes straight to pod IPs without iptables and conntrack
    Action items
    • Move all workloads to ALB IP mode gradually
    Lessons

    Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application. With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.

    View case
  2. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020
    Symptoms

    Prometheus detected a high rate of 5xx responses on three services, called A, B and C in the write-up, and paged the on-call engineers through VictorOps at 22:18 EET. The 5xx rate was also visible in Kibana.

    DNS resolution failuresHigh error rateStale endpoint routing
    Trigger

    Because load in the cluster was low, the CoreDNS autoscaler reduced the number of CoreDNS pods from 3 to 2.

    Root cause
    DNS or network fault
    Component
    CoreDNSkube-proxyconntrack table
    Contributing factor
    Automation amplificationIgnored early signalNo caching
    Detection

    Prometheus alerted on a high 5xx rate on services A, B and C and paged on-call through VictorOps.

    Remediation
    • reverted the latest release of the affected service
    • a regular deploy created new nodes, the CoreDNS autoscaler added CoreDNS pods, and the conntrack table was rewritten
    Action items
    • Disable the autoscaler for CoreDNS
    • Run a DNS cache server in production
    • Monitor conntrack usage
    Lessons

    Scaling CoreDNS down is not free: if kube-proxy fails to clear conntrack entries for the removed pod, clients keep sending DNS queries to it. Known DNS weaknesses in a cluster should be prioritised before they cause an outage, and fixes that only treat consequences leave the underlying conntrack problem in place.

    View case
Check your cluster for this

These 2 cases involve conntrack table and kube-proxy. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

DNS or network fault2 cases

Component
conntrack table2kube-proxy2Cloud load balancer1CoreDNS1Horizontal pod autoscaler1Ingress controller1Pod termination1
Contributing factor
Automation amplification1Unreviewed defaults1Ignored early signal1No caching1
Organisation
Lotte E-commerce1Preply1
  1. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
  2. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020