Incident Library
Applied filters Root causeBreaking change in an upgradeFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
    Symptoms

    Event ingestion and the Honeycomb UI in EU1 became unavailable at 22:00:10 UTC on April 9, 2024, and the status page reported that the EU1 instance was down. Both the ingestion service and the service behind the web UI stopped at roughly the same moment.

    Service outageStale endpoint routing
    Trigger

    A routine system update to the Kubernetes cluster, rolled out earlier on April 9, followed by a regular deployment that replaced pods one at a time.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerCloud load balancer
    Contributing factor
    Monitoring blind spotUnrepresentative stagingChange with a delayed effect
    Detection

    The first status post, saying the EU1 instance was down, went up at 15:08 PDT (22:08 UTC), about 8 minutes after the outage began. There was no alerting on the AWS Load Balancer Controller.

    Remediation
    • Restarted the AWS Load Balancer Controller
    • Rolled back the Kubernetes cluster system update
    Action items
    • Find the exact cause of the failure so the system update can be deployed safely
    • Add monitoring and alerting on the AWS Load Balancer Controller so on-call engineers learn of a failure quickly and can pause deployments
    Lessons

    A rolling deployment can look healthy while the component that wires new pods into the load balancer is broken; the failure only shows when the last old pod goes away. Honeycomb now treats the AWS Load Balancer Controller as a critical component that needs its own alerting, so deployments can be paused before they remove the last pods that still receive traffic. Extensive testing in development and staging did not reveal the bad interaction.

    View case
  2. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
    Symptoms

    About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

    Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag
    Trigger

    An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

    Root cause
    Breaking change in an upgrade
    Component
    CalicoKubernetes API serverAdmission webhooketcd
    Contributing factor
    Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure
    Detection

    The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

    Remediation
    • deleted the Open Policy Agent webhook configurations to stop API server timeouts
    • terminated all worker nodes of the cluster
    • restored the control plane from the pre-upgrade backup onto the first control plane node
    • switched off the cluster autoscaler until the control plane was stable
    • generated new TLS certificates so replacement control plane nodes could join etcd
    • brought traffic back in steps from 1% to 100%
    • restarted legacy services that had been stopped by hand
    Action items
    • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
    • Standardise components across clusters and replace bespoke configurations
    • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
    • Add pre-production clusters that can be tested with confidence
    • Improve the backup restore process and make cluster upgrades safe
    Lessons

    Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

    View case
Check your cluster for this

These 2 cases involve Admission webhook and Calico. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
Admission webhook1Calico1Cloud load balancer1etcd1Ingress controller1Kubernetes API server1
Contributing factor
Monitoring blind spot2Unrepresentative staging2Untested backup1Configuration not in code1Change with a delayed effect1Single point of failure1
Organisation
Honeycomb1Reddit1
  1. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  2. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023