Cases10 cases

1 to 5 of 10

1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed

   MIT Open Learning2026sev1

   Symptoms

   At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

   Service outageTLS certificate errors

   Trigger

   The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

   Root cause

   Configuration change

   Component

   cert-managerCDN and edge networkAuthoritative DNS servers and records

   Contributing factor

   No configuration testsMonitoring blind spotChange with a delayed effect

   Detection

   Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

   Remediation

   • Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate

   Action items

   • Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
   • Deploy the Pulumi update for the xPRO application in Production
   • Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
   • Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
   • Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires

   Lessons

   Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

   View case
2. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters

   IONOS Cloud2026

   Symptoms

   On 8 May 2026 IONOS Cloud reported "K8s Control Planes not available": some Managed Kubernetes customers in DE/TXL, DE/FRA and DE/FRA/2 had trouble connecting to the control plane and saw degraded Kubernetes functionality. Between 15:22 and 16:12 (UTC+2) applications meant for a new internal development cluster were installed on other customer clusters, which were also modified.

   Control plane unresponsiveImage pull failures

   Trigger

   A routine internal operation onboarded a new Managed Kubernetes development cluster. The scoping configuration file for the onboarding operator failed to reach the deployment branch because of a merge conflict, and at the same time a second change removed the operator from the exclusion list, which deployed it at once.

   Root cause

   Faulty deployment

   Component

   CI systemContainer registry and its credentials

   Contributing factor

   No configuration testsExcessive blast radiusMissed in review

   Remediation

   • Corrected the onboarding operator so it needs an explicit namespace configuration before acting on any cluster
   • Reconfigured the deployment system to fail deployments when required configuration files are missing
   • Rolled back the identified changes on the affected resources
   • Ran a recovery script, validated in staging, that removed all unintended installations and updates from the affected clusters

   Action items

   • Add a mandatory CI/CD step that confirms all required configuration files reached the target branch before any deployment starts
   • Require additional peer review and change management approval for every deployment that touches production, whatever its size
   • Give teams a way to inform each affected customer in time about the status of post-incident cleanup

   Lessons

   IONOS treats the unintended change to cluster configurations as a failure of the controls that should keep internal operations from affecting customer environments.

   View case
3. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device

   UK Ministry of Justice2023

   Symptoms

   At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

   Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion

   Trigger

   Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

   Root cause

   Configuration change

   Component

   Node root diskNode launch templateNode poolAWS VPC CNICoreDNS

   Contributing factor

   Missed in reviewNo configuration testsNo alert on limit

   Detection

   The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

   Remediation

   • Uncordoned the old default node group and drained the new nodes to move workloads back
   • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
   • Drained the old node group onto the fixed nodes

   Action items

   • Update the runbook to compare launch template changes during an EKS module upgrade
   • Create a test setup that pulls images of different sizes, like live
   • Update the RootVolUtilisation alert runbook to check the disk space configuration
   • Scale CoreDNS dynamically based on the number of nodes
   • Investigate whether IPv6 can solve the IP prefix starvation problem
   • Add drift testing to identify when a terraform plan shows a change to the launch template
   • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation

   Lessons

   The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

   View case
4. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed

   Wikimedia Foundation2023

   Symptoms

   On 9 August 2023, right after a MediaWiki configuration deployment at 11:41 UTC, the MediaWiki deployments on Kubernetes terminated TLS with a wrong certificate, and about 1% of Wikimedia web and API traffic, around 120 requests per second, received errors for about 30 minutes. The issue was detected by a page for failed service probes on mw-api-ext and mw-api-int.

   High error rateTLS certificate errorsControl plane lag

   Trigger

   At 09:37 a change to the global Kubernetes defaults was merged that switched supported charts to cert-manager certificates. The next MediaWiki on Kubernetes deployment, a configuration change at 11:41, picked it up.

   Root cause

   Configuration change

   Component

   cert-managerHelm

   Contributing factor

   Change with a delayed effectNo configuration tests

   Detection

   A page from the ProbeDown alerts for mw-api-ext:4447 and mw-api-int:4446 at 11:47; a KubernetesAPILatency alert had fired at 11:46.

   Remediation

   • Identified the issue and prepared a patch, https://gerrit.wikimedia.org/r/947331
   • Redeployed the MediaWiki deployments on Kubernetes

   Action items

   • Review whether deployment procedures and testing should be strengthened, for example against surprising changes on the next deployment and with canary deployments for Kubernetes
   • Look at metrics that become unavailable or unhealthy during deployments, or at reducing the impact of deployments

   Lessons

   The cause was found quickly because the person who made the breaking change was responding. The change should have been caught by CI, but the repository it was made in (hieradata) does not run the CI of the deployment charts. Only 1% of traffic going to Kubernetes kept the impact small, and the low error volume made it slower to link the errors to Kubernetes.

   View case
5. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating

   UK Ministry of Justice2023

   Symptoms

   At 08:15 a user posted in the #ask-cloud-platform channel that new deployments were not starting: their pods stayed in ContainerCreating.

   Pods stuck in ContainerCreatingIP address exhaustion

   Root cause

   Configuration change

   Component

   AWS VPC CNINode pool

   Contributing factor

   No configuration tests

   Detection

   A user reported new deployments stuck in ContainerCreating in the #ask-cloud-platform channel at 08:15; the incident was declared at 09:31.

   Remediation

   • Added new nodes and cordoned and drained the six affected nodes
   • Enabled prefix delegation in the VPC CNI on the live cluster after comparing its CNI settings with a 1.23 test cluster

   Action items

   • Add a test or check that confirms IP address allocation is working as expected (#4669)

   Lessons

   The write-up states no lessons beyond its review action, which is to add a check that IP address allocation works.

   View case

By root cause3