Incident Library
Applied filters Root causeConfiguration changeSymptomHigh latencyClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases7 cases

4 to 7 of 7

  1. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
    Symptoms

    At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.

    High error rateHigh latencyIntermittent timeouts
    Trigger

    A k8s-workloads/gitlab-com pipeline deployed merge request !1643, a GitLab chart bump that changed the port names of Kubernetes Services and Deployments (related to the Prometheus service and pod monitors), starting with the canary environment gprd-cny at 07:49 UTC.

    Root cause
    Configuration change
    Component
    HelmCloud load balancerHAProxy
    Contributing factor
    Automation amplification
    Detection

    A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.

    Remediation
    • drained the canary environment and re-enabled it after twenty minutes
    • cancelled the deployment jobs for clusters c and d before they started
    • drained gprd-us-east1-d in HAProxy and re-enabled it five minutes later
    • raised the minimum replica count above 100 (merge request !1650)
    • drained gprd-us-east1-c in HAProxy, deployed the change to it, and re-enabled it
    • reverted the minimum replica increase (merge request !1652)
    Action items
    • Implement graceful draining of a whole regional cluster
    • Create a Chatops command that puts PagerDuty into maintenance mode
    Lessons

    Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes. GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.

    View case
  2. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
    Symptoms

    The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.

    High latencyAutoscaler misbehaviour
    Trigger

    Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.

    Root cause
    Configuration change
    Component
    Horizontal pod autoscaler
    Contributing factor
    Unverified assumption
    Detection

    The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.

    Remediation
    • Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Action items
    • Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Lessons

    When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.

    View case
  3. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019
    Symptoms

    After a Spring-based JVM microservice was deployed to Kubernetes and part of production traffic was routed to it, request latency there was up to 10 times higher than on EC2. Behind the NGINX ingress controller the backend took 100 to 200 ms to answer, where the EC2 instances took about 20 ms.

    High latency
    Trigger

    Moving the service onto the Kubernetes platform, where KIAM served IAM credentials to pods with its default 15-minute lifetime.

    Root cause
    Configuration change
    Component
    Credential broker for pods
    Contributing factor
    Unreviewed defaults
    Detection

    After part of production traffic was routed to the Kubernetes deployment, its request latency was seen to be up to 10 times higher than on EC2.

    Remediation
    • reconfigured KIAM to request IAM credentials with a longer expiration period
    Lessons

    KIAM's short credential lifetime for pods and the AWS Java SDK's eager refresh of any credential within 15 minutes of expiry are each reasonable, but together they force a metadata round trip on every request. When latency rises after a move to Kubernetes, the cause is often the interaction of components that never ran together before rather than Kubernetes itself. Here a packet capture from inside the container found what latency metrics and DNS tests had not.

    View case
  4. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019
    Symptoms

    The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

    Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled
    Trigger

    A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

    Root cause
    Configuration change
    Component
    Container memory limitkubeletPrometheus
    Contributing factor
    No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius
    Detection

    Users reported error pages, slow responses and timeouts from several services; no alert fired.

    Remediation
    • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
    • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
    • fixed the ElastAlert problem that caused the memory growth
    Action items
    • Define alerts on resource usage such as CPU, memory and disk space
    • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
    • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
    • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
    • Set resource limits for ElastAlert through the Helm values
    • Configure namespaces with default memory requests and limits
    • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
    • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
    • Deploy through a pipeline that includes a smoke test step
    • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics
    Lessons

    Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

    View case
Check your cluster for this

These 7 cases involve Cloud load balancer and HAProxy. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change7 cases

Component
Cloud load balancer2HAProxy2Container memory limit2In-memory cache1conntrack table1Credential broker for pods1Helm1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1kubelet1Persistent volumes and stateful sets1Prometheus1
Contributing factor
Unverified assumption3Excessive blast radius3Unreviewed defaults2Retry storm2Automation amplification1Change with a delayed effect1Limited debug access1Monitoring blind spot1No alert on limit1No health probes1No resource requests or limits1No headroom1Single point of failure1
Organisation
GitLab2Adevinta1AdGuard VPN1LINE+1NU.nl1OpenAI1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  3. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  4. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  5. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
  6. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019
  7. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019