Incident Library
Applied filters FactorChange with a delayed effectDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases22 cases

2 to 6 of 22

  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026sev1
    Symptoms

    At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

    Service outageTLS certificate errors
    Trigger

    The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

    Root cause
    Configuration change
    Component
    cert-managerCDN and edge networkAuthoritative DNS servers and records
    Contributing factor
    No configuration testsMonitoring blind spotChange with a delayed effect
    Detection

    Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

    Remediation
    • Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate
    Action items
    • Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
    • Deploy the Pulumi update for the xPRO application in Production
    • Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
    • Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
    • Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires
    Lessons

    Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

    View case
  2. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026
    Symptoms

    An alert fired one day because an application's error rate had suddenly risen. The error log of the Aurora MySQL RDS instance for that time showed Aurora killing connections and declining queries to avoid running out of memory. The killed queries also made application latency worse.

    High error rateMemory pressure
    Trigger

    An application release added a batch job run every minute by a Kubernetes CronJob. Each run's short-lived pod connected to the database from a new pod IP.

    Root cause
    Capacity exhaustion
    Component
    DatabaseJobs and CronJobs
    Contributing factor
    Unreviewed defaultsChange with a delayed effect
    Detection

    An alert fired on the application's error rate.

    Remediation
    • Set performance_schema to manual management and capped two related Performance Schema parameters, unlimited by default, at 100 in the parameter group, then restarted the RDS instance
    Lessons

    When system alerts fire but the database's load metrics look calm, Performance Schema growth is worth suspecting. The two parameters that were capped are unlimited by default, so connections from ever-changing source addresses keep adding to the records. Capping them has a cost: once the cap is exceeded, per-host figures in Performance Insights are missing. Putting RDS Proxy between the application and Aurora MySQL to keep source hosts stable is another possible measure.

    View case
  3. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
    Symptoms

    An attempt to launch a new hub failed because its ingress could not obtain a new TLS certificate. The day before, the same engineer had recreated the support deployment on the 2i2c-aws-us cluster and seen its TLS certificate orders stuck pending.

    Misrouted requestsDeploys failing
    Trigger

    Creating new hub deployments that needed new TLS certificates on clusters recently migrated from ingress-nginx to nginx-ingress.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllercert-manager
    Contributing factor
    Change with a delayed effect
    Detection

    An attempt to launch a new hub failed; the engineer had seen certificate orders stuck pending on another cluster the day before and declared an incident.

    Remediation
    • annotated the Ingress objects that need TLS so that cert-manager could edit them in place to complete the ACME challenge
    Action items
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/8016
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/3278
    Lessons

    Swapping one ingress controller for another can change how Ingress objects that share a host are combined. cert-manager's temporary challenge Ingress depends on that behaviour, so the breakage shows up when a deployment needs a new certificate. 2i2c notes that it got lucky: it had already learned of this possible cause while testing on an internal cluster, and changing the cert-manager configuration was quick.

    View case
  4. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
    Symptoms

    From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

    Service outageTLS certificate errors
    Trigger

    The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer
    Contributing factor
    Change with a delayed effectMonitoring blind spot
    Detection

    Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

    Remediation
    • Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
    • Redeployed the AWS Load Balancer Controller configuration with Helm
    • Verified that new certificates were generated and the webhooks worked again
    Action items
    • Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
    • Add high-level application access alerting for each SaaS cluster from an external probe
    View case
  5. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025
    Symptoms

    From about 02:29 UTC on 24 September 2025, customers of Linode Kubernetes Engine (LKE) in US-Southeast (Atlanta) could not manage their clusters, and autoscaling was impacted. Internal logs showed 5xx errors whenever customers tried to manage LKE clusters associated with the affected cluster controller.

    High error rateAutoscaler misbehaviour
    Trigger

    Old certificates on the LKE cluster controllers were revoked at 02:29 UTC on 24 September 2025, after the expiring certificates had been rotated on 23 September.

    Root cause
    Certificate or credential expiry
    Component
    Managed Kubernetes control plane and provider APICluster autoscaler
    Contributing factor
    Change with a delayed effect
    Remediation
    • deployed a fix for the certificate key mismatch
    • restarted the API endpoints
    Action items
    • Continue investigating the root cause and take preventive actions
    View case
Check your cluster for this

These 22 cases involve Ingress controller and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Configuration change10 cases

Component
cert-manager2Cilium2Cloud load balancer2conntrack table2Ingress controller2Linux kernel2CDN and edge network1Configuration management tool1Configuration templating1ConfigMaps and Secrets1CoreDNS1Client-side DNS resolver and cache1Authoritative DNS servers and records1etcd1external-dns1HAProxy1Helm1HTTP and RPC client library1Karpenter1Kubernetes API server1kubelet1Node operating system image1Node pool1
Contributing factor
Change with a delayed effect10Unverified assumption3Retry storm3Unreviewed defaults2Monitoring blind spot2No configuration tests2Excessive blast radius1Configuration not in code1Ignored early signal1Unknown limit1Limited debug access1Harmful manual action1No alert on limit1No runbook1Shared credential1
Organisation
Wikimedia Foundation2Adyen1Datadog1DigitalOcean1LINE Plus1MIT Open Learning1Pusher1Python Software Foundation1Stytch1
  1. IL-0246 New pods on Adyen's big data Kubernetes nodes could not start, failing with "Cilium API client timeout exceeded"
    Adyen2026
  2. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  3. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  4. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023
  5. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023
  6. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022
  7. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE Plus2022
  8. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
  9. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019
  10. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019

Capacity exhaustion2 cases

Component
Jobs and CronJobs2Calico1Database1Helm1
Contributing factor
Change with a delayed effect2Unreviewed defaults1No runbook1
Organisation
CyberAgent1Wikimedia Foundation1
  1. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026
  2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024

Breaking change in an upgrade5 cases

Component
Ingress controller3cert-manager2Cloud load balancer2Node operating system image2Admission webhook1Cilium1ConfigMaps and Secrets1Helm1Node pool1systemd-networkd1
Contributing factor
Change with a delayed effect5Monitoring blind spot2Automation amplification1Unreviewed defaults1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
2i2c1Datadog1Enigmo1Honeycomb1PortSwigger1
  1. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  2. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  3. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  4. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  5. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable
    Enigmo2020

Certificate or credential expiry1 case

Component
Cluster autoscalerManaged Kubernetes control plane and provider API
Contributing factor
Change with a delayed effect
Organisation
Linode
  1. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025

Software defect3 cases

Component
Admission webhook1Cloud load balancer1GPU driver1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Linux kernel1
Contributing factor
Change with a delayed effect3Ignored early signal2Monitoring blind spot2Excessive blast radius1No configuration tests1Single point of failure1
Organisation
Airbnb1Barush Mendez1DFDS1
  1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  2. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  3. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020

Faulty deployment1 case

Component
CI systemNode pool
Contributing factor
Unverified assumptionChange started before off-hoursChange with a delayed effectNo alert on limitSingle point of failure
Organisation
Kubernetes project
  1. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019