Incident Library
Applied filters ComponentApplication databaseSymptomHigh error rateOrganisationGitLabClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
    Symptoms

    GitLab's monitoring and alerting fired at once. One of the alerts reported a Patroni failover, and users of GitLab.com received 5XX errors; the CDN counted about 13 million requests answered with 503 during the outage.

    Service outageHigh error rateResources deleted
    Trigger

    An engineer ran a routine task with GitLab's tools in a local development environment; during a local deploy or rollback operation the tool's context changed from local development to production, and the change was applied to production.

    Root cause
    Operator error
    Component
    Application database
    Contributing factor
    No safeguard on destructive change
    Detection

    GitLab's monitoring and alerting fired immediately; one of the alerts reported a Patroni failover.

    Remediation
    • re-applied the Consul release to the production Kubernetes cluster, with manual steps because the tooling did not support the redeploy directly
    Action items
    • Improve the Kubernetes deployment process to reduce the risk of running harmful deployment operations against production from local environments (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16875)
    • Integrate Teleport with the Kubernetes clusters, which could give SREs a read-only role by default (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16221)
    Lessons

    A tool that can switch its target from a local environment to production during a deploy or rollback turns routine local work into a production change. GitLab plans to improve its Kubernetes deployment process so that harmful deployment operations are less likely to be run against production from local environments, and to integrate Teleport with its clusters, which could give SREs a read-only role by default. Detection and diagnosis were fast because the alerts led straight from Patroni to its dependency, Consul. Bringing the deleted release back needed manual work, because the tooling did not make the redeploy straightforward, but it was still done relatively quickly.

    View case
  2. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
    Symptoms

    On 2022-04-25 the Sidekiq Apdex (the share of background jobs that finish within a target time) dropped, and an engineer declared an incident in Slack at 15:13 UTC. Users saw intermittent slowness across GitLab.com, and asynchronous work such as pipeline creation and import and export took unusually long.

    High latencyDatabase overloadOOMKilledHigh error rate
    Trigger

    The Consul agent in GitLab.com's GKE pods had memory demand spikes that did not fit in the pods' 500MB memory limit, and the agent was killed again and again.

    Root cause
    Capacity exhaustion
    Component
    Container memory limitService discoveryApplication database
    Contributing factor
    –
    Detection

    An engineer declared the incident in Slack at 15:13 UTC after the Sidekiq Apdex dropped; the second event was found when monitoring detected an elevated error rate at 01:21 UTC on 2022-04-26.

    Remediation
    • planned at 16:13 to raise the per-user connection limit for the gitlab database user and the pgbouncer backend pools for the main database; changes were applied at 17:26 and confirmed at 17:47
    • raised the pod memory limits from 500MB to 1GB so the Consul agents survive memory spikes
    Action items
    • Monitor the Consul agents on GKE pods to see whether the 1GB memory limit keeps them healthy
    • Investigate the Consul memory demand spikes and what can be done beyond raising the hard memory limit
    Lessons

    When the local Consul agent that Rails uses for database service discovery keeps dying, Rails does not fail; it sends every query to the last known primary. What engineers saw was database saturation and slow background jobs, which pointed the investigation first at a recent pgbouncer pool change. The memory limit of the Consul agent in every pod therefore decided the load on the primary database.

    View case
Check your cluster for this

These 2 cases involve Application database and Container memory limit. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Operator error1 case

Component
Application database
Contributing factor
No safeguard on destructive change
Organisation
GitLab
  1. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022

Capacity exhaustion1 case

Component
Application databaseContainer memory limitService discovery
Contributing factor
–
Organisation
GitLab
  1. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022