Incident Library
Applied filters Root causeConfiguration changeComponentHorizontal pod autoscalerClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
    Symptoms

    The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.

    High latencyAutoscaler misbehaviour
    Trigger

    Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.

    Root cause
    Configuration change
    Component
    Horizontal pod autoscaler
    Contributing factor
    Unverified assumption
    Detection

    The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.

    Remediation
    • Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Action items
    • Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Lessons

    When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.

    View case
Check your cluster for this

This case involves Horizontal pod autoscaler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Horizontal pod autoscaler
Contributing factor
Unverified assumption
Organisation
GitLab
  1. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021