Incident Library
Applied filters Root causeConfiguration changeComponentCloud load balancerSymptomHigh error rateClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
    Symptoms

    At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.

    High error rateHigh latencyIntermittent timeouts
    Trigger

    A k8s-workloads/gitlab-com pipeline deployed merge request !1643, a GitLab chart bump that changed the port names of Kubernetes Services and Deployments (related to the Prometheus service and pod monitors), starting with the canary environment gprd-cny at 07:49 UTC.

    Root cause
    Configuration change
    Component
    HelmCloud load balancerHAProxy
    Contributing factor
    Automation amplification
    Detection

    A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.

    Remediation
    • drained the canary environment and re-enabled it after twenty minutes
    • cancelled the deployment jobs for clusters c and d before they started
    • drained gprd-us-east1-d in HAProxy and re-enabled it five minutes later
    • raised the minimum replica count above 100 (merge request !1650)
    • drained gprd-us-east1-c in HAProxy, deployed the change to it, and re-enabled it
    • reverted the minimum replica increase (merge request !1652)
    Action items
    • Implement graceful draining of a whole regional cluster
    • Create a Chatops command that puts PagerDuty into maintenance mode
    Lessons

    Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes. GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.

    View case
  2. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
    Symptoms

    At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.

    Service outageHigh error rate
    Trigger

    The pipeline replaced the AWS VPC CNI as part of a refactoring that moved EKS add-ons from kubectl to native Terraform management.

    Root cause
    Configuration change
    Component
    AWS VPC CNICloud load balancerTerraform
    Contributing factor
    Unverified assumptionNo configuration tests
    Detection

    Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.

    Remediation
    • committed a fix that reverted the infrastructure-modules reference
    • re-triggered the pipeline after it failed on an AWS API timeout
    • let the pipeline reattach the target groups to the ALB
    Action items
    • Revert the ALB de-registration delay to the default of 300 seconds
    • Update the roll-over script to set the ALB de-registration delay to zero temporarily before a roll-over
    • Create an ingress test suite
    Lessons

    The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.

    View case
Check your cluster for this

These 2 cases involve Cloud load balancer and HAProxy. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Cloud load balancer2HAProxy1Helm1Terraform1AWS VPC CNI1
Contributing factor
Unverified assumption1Automation amplification1No configuration tests1
Organisation
DFDS1GitLab1
  1. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  2. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021