IL-0522 Jenkins project · 2021-11-18kubernetesnetworking

An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours

Fastly serves jenkins.io from an Ingress in the Jenkins infrastructure Kubernetes cluster. At 10:42 UTC on 18 November 2021 the ingress-nginx Helm chart was upgraded from 4.0.6 to the patch version 4.0.8, which brought a new admission webhook rule forbidding Ingress annotations that contain an Nginx configuration block. The jenkins.io Ingress rule did not comply and was deleted, the cluster answered HTTP/404 to Fastly, and from 11:07 UTC Fastly dropped pages from its cache and served HTTP/503. A user reported the errors at 11:32 UTC; no monitoring covered origin.jenkins.io. The team had trusted the patch version of a chart for a sensitive component, the Ingress controller, without a deep read of its changelog.

At 13:40 UTC the team disabled its automated systems, rolled the chart back to 4.0.6 and deleted the admission webhook by hand, which ended the first outage after 2h 37min. At 15:30 UTC, after a first batch of changes to the jenkins.io configuration, the chart was deployed at 4.0.8 again and caused a second outage. The fixes removed the forbidden annotations and upgraded the jenkins.io Helm chart to the last version known to work with ingress-nginx 4.0.8. At 17:11 UTC all required fixes had been applied, the automated system redeployed the jenkins.io Helm chart, the website was back and Fastly was told to drop all cached pages, ending the second outage after 1h 41min. The upgrade to ingress-nginx 4.0.9 was later done after a deep changelog check.

Severity: –
Scope: jenkins.io (also www.jenkins.io) was unavailable and answered HTTP/503 from 11:07 to 13:40 UTC and from 15:30 to 17:11 UTC on 18 November 2021
Users affected: –
Region: –
Time to detect: 25m
Time to resolve: 6h 4m

Symptoms

At 11:32 UTC on 18 November 2021, a user reported in the jenkinsci/jenkins Gitter channel that https://www.jenkins.io/doc/pipeline/steps/ was answering HTTP/503 errors with the message "Guru meditation" from Fastly's varnish.

The failure had started with an upgrade of the ingress-nginx Helm chart from 4.0.6 to 4.0.8, a patch version, at 10:42 UTC. The new chart created an admission webhook rule that forbids any Ingress annotation containing an Nginx configuration block, such as any `{}`. The Ingress rule for jenkins.io, www.jenkins.io, origin.jenkins.io and www.origin.jenkins.io did not comply and was deleted, so the pages answered HTTP/404 to Fastly.

From 11:07 UTC Fastly began to drop pages of jenkins.io from its cache because it could not update them from the Kubernetes cluster, and jenkins.io answered HTTP/503. There was no monitoring on origin.jenkins.io to report the errors once they were visible, and status.jenkins.io reported the incident at 11:49 UTC.

A manual fix brought jenkins.io back at 13:40 UTC. At 15:30 UTC, after a first batch of changes to the jenkins.io configuration in Kubernetes, the ingress-nginx chart was deployed at 4.0.8 again and jenkins.io went down a second time, until 17:11 UTC.

Symptom Service outage Resources deleted

Root cause

Root cause Breaking change in an upgrade

Component Ingress controller Admission webhook Helm CDN and edge network

Contributing factor Unverified assumption Cache-masked failure Monitoring blind spot Harmful manual action

Trigger: The ingress-nginx Helm chart was upgraded from 4.0.6 to 4.0.8 at 10:42 UTC, and the new chart added an admission webhook rule that forbids Nginx configuration blocks in Ingress annotations.

Detection

A user reported in the jenkinsci/jenkins Gitter channel at 11:32 UTC that a jenkins.io page answered HTTP/503 errors, and status.jenkins.io reported the incident at 11:49 UTC.

Remediation

Disabled the automated systems, rolled the ingress-nginx chart back to 4.0.6 and deleted the admission webhook by hand
Removed the forbidden annotations from the jenkins.io configuration
Upgraded the jenkins.io Helm chart to the last version known to work with ingress-nginx 4.0.8
Redeployed the jenkins.io Helm chart through the automated system and told Fastly to drop all cached pages

Action items

Always read the changelog in depth when updating the ingress controllers, with two people pairing in real time
Add custom Datadog monitoring for www.origin.jenkins.io so that such an outage is detected
Make the monitor send the Host header www.jenkins.io so it is not redirected to Fastly
Check a set of pages, such as the pipeline steps page, not only the front page
Consider an ephemeral k3s end-to-end test before deploying to production

Lessons

Trusting a patch version of a Helm chart for the Ingress controller without reading its changelog in depth let a new admission rule remove the jenkins.io Ingress in production. Without monitoring on origin.jenkins.io, a user found the outage. A complete staging cluster with the same DNS setup would have caught the problem before production.

Sources

https://github.com/jenkins-infra/documentation/blob/main/postmortem/20211118-jenkinsio-outage.md · github

References

Related cases

IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS downPortSwigger · 2026
same root cause · shared Helm, Ingress controller, Admission webhook
IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and changeUK Ministry of Justice · 2021
same root cause · shared Ingress controller, Admission webhook
IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates2i2c · 2026
same root cause · shared Ingress controller
IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applicationsUK Ministry of Justice · 2026
same root cause · shared Ingress controller
IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024
same root cause · shared Ingress controller
IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replacedHoneycomb · 2024
same root cause · shared Ingress controller
IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in productionSunwoo Park · 2023
same root cause · shared Admission webhook
IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023
same root cause · shared Admission webhook

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Ingress controller, Admission webhook, Helm among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0522
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0522/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.