Incident Library

Applied filters Root causeConfiguration changeDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

5 results

Lit on the graph.

GitLab, 2021
DFDS, 2021
DFDS, 2020
DFDS, 2020
Adevinta, 2019

Cases46 cases

34 to 38 of 46

IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
GitLab2021
Symptoms
The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.
High latency Autoscaler misbehaviour
Trigger
Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.
Root cause
Configuration change
Component
Horizontal pod autoscaler
Contributing factor
Unverified assumption
Detection
The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.
Remediation
Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
Action items
Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
Lessons
When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.
View case
IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
DFDS2021
Symptoms
At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.
Service outage High error rate
Trigger
The pipeline replaced the AWS VPC CNI as part of a refactoring that moved EKS add-ons from kubectl to native Terraform management.
Root cause
Configuration change
Component
AWS VPC CNI Cloud load balancer Terraform
Contributing factor
Unverified assumption No configuration tests
Detection
Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.
Remediation
committed a fix that reverted the infrastructure-modules reference
re-triggered the pipeline after it failed on an AWS API timeout
let the pipeline reattach the target groups to the ALB
Action items
Revert the ALB de-registration delay to the default of 300 seconds
Update the roll-over script to set the ALB de-registration delay to zero temporarily before a roll-over
Create an ingress test suite
Lessons
The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.
View case
IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
DFDS2020
Symptoms
At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.
Deploys failing Control plane lag
Trigger
During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.
Root cause
Configuration change
Component
Admission webhook Kubernetes API server
Contributing factor
No configuration tests Unverified assumption Monitoring blind spot
Detection
Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.
Remediation
toggled the Alcide admission controller to verify that it caused the timeouts
permanently deleted the Alcide admission controller
Action items
Monitor Kubernetes deployments for timeouts
Monitor 5xx status codes from the Kubernetes API server
Lessons
New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.
View case
IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
DFDS2020
Symptoms
On 9 June 2020 at 12:25 CET a CloudWatch alert fired on a high count of 5XX responses from the AWS Application Load Balancers (ALBs) of DFDS's Hellman Kubernetes cluster, and at 12:28 the SRE team verified that no ingress traffic was reaching the cluster. Every service running inside the cluster was unreachable from outside it, because both ALBs, the authenticated one and the one without authentication, no longer had any Kubernetes worker nodes attached.
Service outage
Trigger
Planned maintenance to split the Hellman cluster's auto-scaling groups by availability zone applied Terraform changes at 12:20 CET that removed the ALB instance attachments.
Root cause
Configuration change
Component
Terraform Cloud load balancer Node launch template
Contributing factor
Unrepresentative staging Signal buried in noise Monitoring blind spot
Detection
A CloudWatch alert detected a high count of 5XX responses for the ALBs, and the lack of ingress traffic was then verified.
Remediation
Attached the worker node instances to the ALBs by hand
Implemented a permanent fix within the hour
Action items
Implement alerts on healthy and unhealthy ALB targets for more precise and immediate notice
Reduce alert noise further
Implement host name resolution for Goldpinger
Lessons
General alert noise is still too high, there was no alert on healthy and unhealthy ALB targets, and test clusters should check ingress through the ALBs.
View case
IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
Adevinta2019
Symptoms
After a Spring-based JVM microservice was deployed to Kubernetes and part of production traffic was routed to it, request latency there was up to 10 times higher than on EC2. Behind the NGINX ingress controller the backend took 100 to 200 ms to answer, where the EC2 instances took about 20 ms.
High latency
Trigger
Moving the service onto the Kubernetes platform, where KIAM served IAM credentials to pods with its default 15-minute lifetime.
Root cause
Configuration change
Component
Credential broker for pods
Contributing factor
Unreviewed defaults
Detection
After part of production traffic was routed to the Kubernetes deployment, its request latency was seen to be up to 10 times higher than on EC2.
Remediation
reconfigured KIAM to request IAM credentials with a longer expiration period
Lessons
KIAM's short credential lifetime for pods and the AWS Java SDK's eager refresh of any credential within 15 minutes of expiry are each reasonable, but together they force a metadata round trip on every request. When latency rises after a move to Kubernetes, the cause is often the interaction of components that never ran together before rather than Kubernetes itself. Here a packet capture from inside the container found what latency metrics and DNS tests had not.
View case

Check your cluster for this

These 46 cases involve Cloud load balancer and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change46 cases

Component: Cloud load balancer9 Node pool8 Kubernetes API server7 Configuration templating5 Terraform5 Ingress controller4 AWS VPC CNI4 etcd3 kubelet3 Node launch template3 Container memory limit3 Telemetry agent3 Admission webhook2 Argo CD2 cert-manager2 Cilium2 CNI plugin2 conntrack table2 Container runtime2 CoreDNS2 Client-side DNS resolver and cache2 GKE control plane2 HAProxy2 Helm2 Linux kernel2 Persistent volumes and stateful sets2 Pod priority and preemption2 In-memory cache1 CDN and edge network1 CI system1 Cluster dashboard1 Configuration management tool1 ConfigMaps and Secrets1 Coordination and key-value store1 Credential broker for pods1 Database1 Database proxy and connection pooler1 Authoritative DNS servers and records1 external-dns1 Horizontal pod autoscaler1 HTTP and RPC client library1 Jobs and CronJobs1 Karpenter1 Application logging library1 Node root disk1 Node operating system image1 Node iptables rules1 Pod termination1 Prometheus1 RBAC and service accounts1 Container registry and its credentials1 Service discovery1
Contributing factor: Monitoring blind spot13 Unverified assumption12 Excessive blast radius11 Change with a delayed effect10 Automation amplification8 Unreviewed defaults8 No configuration tests8 Retry storm6 Missed in review6 No safeguard on destructive change5 No staged rollout5 Unrepresentative staging5 No runbook4 Ignored early signal3 Limited debug access3 No alert on limit3 Unknown limit2 Harmful manual action2 No health probes2 No headroom2 Signal buried in noise1 Untested backup1 Configuration not in code1 Cache-masked failure1 No resource requests or limits1 Overlapping failures1 Partial rollout1 Shared credential1 Single point of failure1 Unpinned version1
Organisation: DFDS5 OpenAI4 Grafana Labs3 UK Ministry of Justice3 DigitalOcean2 GitLab2 Wikimedia Foundation2 Adevinta1 AdGuard VPN1 Adyen1 Aqua Security1 CoW Protocol1 Datadog1 Discord1 GitHub1 Google Cloud1 Hans H. Kim1 InfluxData1 Jenkins project1 Jetstack1 JW Player1 LIFULL1 LINE Plus1 MIT Open Learning1 MUSINSA1 NU.nl1 Pusher1 Python Software Foundation1 Skyscanner1 Spotify1 Stytch1 Target1