Incident Library

Applied filters Root causeBreaking change in an upgradeFactorMonitoring blind spotDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

5 results

Lit on the graph.

PortSwigger, 2026
Honeycomb, 2024
Reddit, 2023
CircleCI, 2023
Jenkins project, 2021

Cases5 cases

1 to 5 of 5

IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
PortSwigger2026
Symptoms
From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.
Service outage TLS certificate errors
Trigger
The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.
Root cause
Breaking change in an upgrade
Component
Ingress controller Admission webhook cert-manager Helm ConfigMaps and Secrets Cloud load balancer
Contributing factor
Change with a delayed effect Monitoring blind spot
Detection
Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.
Remediation
Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
Redeployed the AWS Load Balancer Controller configuration with Helm
Verified that new certificates were generated and the webhooks worked again
Action items
Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
Add high-level application access alerting for each SaaS cluster from an external probe
View case
IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
Honeycomb2024
Symptoms
Event ingestion and the Honeycomb UI in EU1 became unavailable at 22:00:10 UTC on April 9, 2024, and the status page reported that the EU1 instance was down. Both the ingestion service and the service behind the web UI stopped at roughly the same moment.
Service outage Stale endpoint routing
Trigger
A routine system update to the Kubernetes cluster, rolled out earlier on April 9, followed by a regular deployment that replaced pods one at a time.
Root cause
Breaking change in an upgrade
Component
Ingress controller Cloud load balancer
Contributing factor
Monitoring blind spot Unrepresentative staging Change with a delayed effect
Detection
The first status post, saying the EU1 instance was down, went up at 15:08 PDT (22:08 UTC), about 8 minutes after the outage began. There was no alerting on the AWS Load Balancer Controller.
Remediation
Restarted the AWS Load Balancer Controller
Rolled back the Kubernetes cluster system update
Action items
Find the exact cause of the failure so the system update can be deployed safely
Add monitoring and alerting on the AWS Load Balancer Controller so on-call engineers learn of a failure quickly and can pause deployments
Lessons
A rolling deployment can look healthy while the component that wires new pods into the load balancer is broken; the failure only shows when the last old pod goes away. Honeycomb now treats the AWS Load Balancer Controller as a critical component that needs its own alerting, so deployments can be paused before they remove the last pods that still receive traffic. Extensive testing in development and staging did not reveal the bad interaction.
View case
IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
Reddit2023
Symptoms
About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.
Service outage High error rate DNS resolution failures Monitoring outage Pods stuck in ContainerCreating Control plane lag
Trigger
An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.
Root cause
Breaking change in an upgrade
Component
Calico Kubernetes API server Admission webhook etcd
Contributing factor
Configuration not in code Unrepresentative staging Untested backup Monitoring blind spot Single point of failure
Detection
The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.
Remediation
deleted the Open Policy Agent webhook configurations to stop API server timeouts
terminated all worker nodes of the cluster
restored the control plane from the pre-upgrade backup onto the first control plane node
switched off the cluster autoscaler until the control plane was stable
generated new TLS certificates so replacement control plane nodes could join etcd
brought traffic back in steps from 1% to 100%
restarted legacy services that had been stopped by hand
Action items
Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
Standardise components across clusters and replace bespoke configurations
Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
Add pre-production clusters that can be tested with confidence
Improve the backup restore process and make cluster upgrades safe
Lessons
Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.
View case
IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
CircleCI2023
Symptoms
On 14 March 2023 at about 18:00 UTC, communication errors rose on one CircleCI service, and pipelines for all customers began to start late until jobs did not run at all. Within an hour the errors had spread to many services and to every function of the application, including the CircleCI UI and API.
Service outage Intermittent connection failures DNS resolution failures
Trigger
At about 17:40 UTC on 14 March the infrastructure team resumed the upgrade of the main production Kubernetes cluster with a changed approach for some components, after a first attempt on 13 March had been rolled back. The ongoing upgrade left kubelet and kube-proxy at mismatched versions.
Root cause
Breaking change in an upgrade
Component
kube-proxy kubelet CoreDNS Client-side DNS resolver and cache
Contributing factor
Partial rollout Ineffective rollback Monitoring blind spot Harmful manual action
Detection
At 18:00 UTC engineers saw communication errors rise on one service, then job processing delays climbing and job queues backing up across several services. The kube-proxy "Sync failed" errors were found only in the analysis after the incident.
Remediation
Restarted every node in the cluster so each node built a fresh iptables ruleset
Restarted a separate pool of dependencies so jobs using contexts or the machine executor ran again
Added capacity to work through the backlog of jobs
Restarted the RabbitMQ nodes to fix a few corrupted queues
Action items
Improve the Kubernetes upgrade process to limit the risk of future upgrades
Add metrics and alerts for kube-proxy sync errors
Evaluate ways to make the RabbitMQ infrastructure more resilient
Lessons
No single trigger caused the incident. Because networking underlies everything in the cluster, Kubernetes upgrades carry inherent risk, and CircleCI's risk mitigation did not hold up when several events interacted.
View case
IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
Jenkins project2021
Symptoms
At 11:32 UTC on 18 November 2021, a user reported in the jenkinsci/jenkins Gitter channel that https://www.jenkins.io/doc/pipeline/steps/ was answering HTTP/503 errors with the message "Guru meditation" from Fastly's varnish.
Service outage Resources deleted
Trigger
The ingress-nginx Helm chart was upgraded from 4.0.6 to 4.0.8 at 10:42 UTC, and the new chart added an admission webhook rule that forbids Nginx configuration blocks in Ingress annotations.
Root cause
Breaking change in an upgrade
Component
Ingress controller Admission webhook Helm CDN and edge network
Contributing factor
Unverified assumption Cache-masked failure Monitoring blind spot Harmful manual action
Detection
A user reported in the jenkinsci/jenkins Gitter channel at 11:32 UTC that a jenkins.io page answered HTTP/503 errors, and status.jenkins.io reported the incident at 11:49 UTC.
Remediation
Disabled the automated systems, rolled the ingress-nginx chart back to 4.0.6 and deleted the admission webhook by hand
Removed the forbidden annotations from the jenkins.io configuration
Upgraded the jenkins.io Helm chart to the last version known to work with ingress-nginx 4.0.8
Redeployed the jenkins.io Helm chart through the automated system and told Fastly to drop all cached pages
Action items
Always read the changelog in depth when updating the ingress controllers, with two people pairing in real time
Add custom Datadog monitoring for www.origin.jenkins.io so that such an outage is detected
Make the monitor send the Host header www.jenkins.io so it is not redirected to Fastly
Check a set of pages, such as the pipeline steps page, not only the front page
Consider an ephemeral k3s end-to-end test before deploying to production
Lessons
Trusting a patch version of a Helm chart for the Ingress controller without reading its changelog in depth let a new admission rule remove the jenkins.io Ingress in production. Without monitoring on origin.jenkins.io, a user found the outage. A complete staging cluster with the same DNS setup would have caught the problem before production.
View case

Check your cluster for this

These 5 cases involve Admission webhook and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade5 cases

Component: Admission webhook3 Ingress controller3 Cloud load balancer2 Helm2 Calico1 CDN and edge network1 cert-manager1 ConfigMaps and Secrets1 CoreDNS1 Client-side DNS resolver and cache1 etcd1 Kubernetes API server1 kube-proxy1 kubelet1
Contributing factor: Monitoring blind spot5 Change with a delayed effect2 Harmful manual action2 Unrepresentative staging2 Unverified assumption1 Untested backup1 Configuration not in code1 Cache-masked failure1 Partial rollout1 Ineffective rollback1 Single point of failure1
Organisation: CircleCI1 Honeycomb1 Jenkins project1 PortSwigger1 Reddit1