Cases10 cases

1 to 5 of 10

1. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity

   GitHub2026

   Symptoms

   From 15:05 UTC on 6 August 2026 GitHub Actions workflow runs failed to start, failed partway through or stayed queued for a long time, on both GitHub-hosted and self-hosted runners. Some requests to the Actions REST API returned errors, and some customers saw unexpected rate limiting in their workflows.

   High error rateHigh latency

   Trigger

   A routine deployment to an internal Actions service that processes events and generates Actions jobs replaced pods, and the remaining capacity became saturated.

   Root cause

   Capacity exhaustion

   Component

   CI system

   Contributing factor

   No headroomExcessive blast radiusHarmful manual action

   Remediation

   • Expanded capacity of the affected services
   • Throttled incoming webhook-triggered work
   • Increased processing capacity for the backlog of affected events
   • Deployed changes that stopped runners from repeatedly trying to acquire invalid jobs
   • Deployed a fix for self-hosted runners that were not picking up jobs and enabled it incrementally
   • Paused migrations with GitHub Enterprise Importer
   • Restored webhook-triggered Actions workflows to full throughput
   • Rolled back the mitigation that left some ARC runners offline, and recovered those runners by hand
   • Shared CLI and UI steps for customers to clear jobs that could not be retried or canceled

   Action items

   • Improve deployment and capacity safeguards for the affected services
   • Strengthen monitoring for the conditions that preceded the incident
   • Improve the resiliency and recovery of queued work and runner assignment
   • Add automatic recovery for self-hosted runners in upcoming Runner and ARC releases
   • Reduce the risk of cascading failures and speed up recovery during large Actions disruptions

   View case
2. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters

   IONOS Cloud2026

   Symptoms

   On 8 May 2026 IONOS Cloud reported "K8s Control Planes not available": some Managed Kubernetes customers in DE/TXL, DE/FRA and DE/FRA/2 had trouble connecting to the control plane and saw degraded Kubernetes functionality. Between 15:22 and 16:12 (UTC+2) applications meant for a new internal development cluster were installed on other customer clusters, which were also modified.

   Control plane unresponsiveImage pull failures

   Trigger

   A routine internal operation onboarded a new Managed Kubernetes development cluster. The scoping configuration file for the onboarding operator failed to reach the deployment branch because of a merge conflict, and at the same time a second change removed the operator from the exclusion list, which deployed it at once.

   Root cause

   Faulty deployment

   Component

   CI systemContainer registry and its credentials

   Contributing factor

   No configuration testsExcessive blast radiusMissed in review

   Remediation

   • Corrected the onboarding operator so it needs an explicit namespace configuration before acting on any cluster
   • Reconfigured the deployment system to fail deployments when required configuration files are missing
   • Rolled back the identified changes on the affected resources
   • Ran a recovery script, validated in staging, that removed all unintended installations and updates from the affected clusters

   Action items

   • Add a mandatory CI/CD step that confirms all required configuration files reached the target branch before any deployment starts
   • Require additional peer review and change management approval for every deployment that touches production, whatever its size
   • Give teams a way to inform each affected customer in time about the status of post-incident cleanup

   Lessons

   IONOS treats the unintended change to cluster configurations as a failure of the controls that should keep internal operations from affecting customer environments.

   View case
3. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster

   Wikimedia Foundation2025

   Symptoms

   On 2025-03-12 from 10:58 UTC, alerts for high backend response times came in, and the MediaWiki deployments mw-api-int, mw-parsoid and mw-jobrunner, which serve MediaWiki for non-user-facing purposes, showed high latency and database errors. The External Storage cluster, whose hosts store the compressed text of wiki page revisions, had a sharp rise in load and connections. The same happened again on 2025-03-13 from 11:15 UTC.

   Database overloadHigh latencyHigh error rateLog floodMonitoring outage

   Trigger

   At 10:44 UTC on 12 March a Scap deployment of the PHP 8.1 rollout for mw-api-int, mw-parsoid and mw-jobrunner was run with the flag -Dbuild_mw_container_image:False; the rollout was deployed with Scap again at 11:15 UTC on 13 March.

   Root cause

   Faulty deployment

   Component

   DatabaseIn-memory cacheConfigMaps and SecretsHelmCI system

   Contributing factor

   Unverified assumptionDependence on a warm cacheSignal buried in noise

   Detection

   Alerts for high backend response times started at 10:58 UTC on 12 March, 14 minutes after the Scap deployment; 2 on-call engineers were paged. The MediaWikiMemcachedHighErrorRate alert that pointed at the cause fired only briefly because its log pipeline was overloaded.

   Remediation

   • Deployed a patch to cirrus-streaming-updater that reduced SUP parallelism
   • Reduced the concurrency of the categoryMembershipChange job in changeprop-jobqueue
   • On 12 March, reverted 1126607 and 1126650 and re-ran Scap
   • On 13 March, rolling-restarted changeprop-jobqueue
   • Rolling-restarted mw-api-int
   • Disabled the categorymembership job in changeprop-jobqueue
   • Reverted 1127476 and 1127478
   • Redeployed mw-api-int to pick up the revert
   • Increased the number of mw-api-int replicas
   • Redeployed mw-jobrunner to pick up the revert
   • Redeployed mw-parsoid to pick up the revert

   Action items

   • Reduce the number of messages sent through channel:Memcached during failures
   • Add helm rollback functionality to Scap
   • Decide whether Scap should update helmfile-defaults when -Dbuild_mw_container_image:False is set
   • Consider removing envvars.inc from MediaWiki images
   • Consider "Why" dashboards that give an overview of a group of related components together with relevant links to follow
   • Coordinate incident responders better
   • Be able to work with MediaWiki developers during incidents

   Lessons

   The write-up puts the weight not on the technical root cause but on why the team could not see, even after the 12 March incident, that three MediaWiki deployments were running without Memcached. Alert dashboards showed what was happening, such as many busy PHP workers, but not why; dashboards linking components that affect each other, such as SQLBlobStore keys and External Storage, would have helped. The alert that pointed in the right direction did not stay firing and would probably have been lost among 78 others. The incident coordinator could not tell who was working on what or who had joined or left, the channels were noisy, the external impact was judged more critical than it was, and trace.wikimedia.org was not used although it might have helped.

   View case
4. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline

   UK Ministry of Justice2024

   Symptoms

   On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

   Service outageNetwork unreachableResources deleted

   Trigger

   At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

   Root cause

   Operator error

   Component

   TerraformCI systemVPC route tables and firewall rules

   Contributing factor

   No safeguard on destructive change

   Detection

   High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

   Remediation

   • Ran terraform apply locally to rebuild the subnet route table associations

   Action items

   • Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
   • Find a way to hold queued pull requests while manual infrastructure updates are under way
   • Find ways to isolate the plan and apply tasks of infrastructure pipelines better

   View case
5. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images

   UK Ministry of Justice2023

   Symptoms

   At 14:22 a Cloud Platform team member noticed failed Docker Hub logins. Fetching an image returned: failed to fetch manifest: Head "https://registry-1.docker.io/v2/ministryofjustice/cloud-platform-tools/manifests/2.1": toomanyrequests: too many failed login attempts for username or IP address

   Image pull failures

   Trigger

   The Docker Hub password was changed as part of LastPass remediation activities while the EKS cluster nodes and Concourse still logged in with the previous password.

   Root cause

   Operator error

   Component

   Container registry and its credentialsNode launch templateNode poolCI system

   Contributing factor

   Credential in configurationShared credential

   Detection

   A Cloud Platform team member saw the failed Docker Hub login error at 14:22. The Kuberhealthy daemonset check showed at 14:52 that the cluster could not pull images, and a user reported ErrImagePull in #ask-cloud-platform at 14:54.

   Remediation

   • Changed the Docker Hub password back to the previous value still used by the EKS cluster nodes and Concourse
   • Confirmed with the reporting user that the pod deployed and that cluster image pulls worked again

   Action items

   • Switch from the Docker Hub username and password to a Docker Hub token created specifically for Cloud Platform
   • Rotate the password cleanly out of hours with a graceful recycle of the nodes

   Lessons

   The write-up states no separate lessons. Its review action replaces the Docker Hub username and password with a Docker Hub token created specifically for Cloud Platform.

   View case

By root cause4