IL-0234 IONOS Cloud · 2026-05-08kubernetes

An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters

IONOS Cloud's onboarding operator, part of its platform framework, registers clusters with the deployment system and relies on a scoping configuration file to limit itself to a defined set of target clusters. For the onboarding of a new development cluster this file was to be propagated automatically to the deployment branch, but the propagation failed silently on a merge conflict; the cherry-pick failure was logged but did not stop the pipeline. A second change removed the operator from the exclusion list and triggered an immediate deployment. The deployment system did not halt the sync when configuration was missing, so the operator ran without any scope, treated further cluster-api based clusters as valid onboarding targets, modified them and installed the development cluster's applications on them. IONOS names a bug in the onboarding operator and, as the root cause, a gap in the CI/CD pipeline that let a deployment proceed without checking that all required configuration files had reached the target branch.

IONOS identified a defect in a configuration filtering mechanism by 15:52 UTC, corrected the onboarding operator and hardened the deployment system configuration, audited the affected resources and rolled back the identified changes by 20:42 UTC on 8 May. A recovery script, tested through a full staging cycle first, then removed all unintended installations and updates from the affected production clusters, finishing on 15 May 2026. Customers were not kept properly informed about this cleanup, which added to their uncertainty.

Severity: –
Scope: Additional Managed Kubernetes clusters were modified and received applications meant for a development cluster; some customers had control plane connection problems and degraded Kubernetes functionality, and cleanup ran until 15 May 2026
Users affected: Some Managed Kubernetes customers
Region: DE/TXL, DE/FRA and DE/FRA/2
Time to detect: –
Time to resolve: –

Symptoms

On 8 May 2026 IONOS Cloud reported "K8s Control Planes not available": some Managed Kubernetes customers in DE/TXL, DE/FRA and DE/FRA/2 had trouble connecting to the control plane and saw degraded Kubernetes functionality. Between 15:22 and 16:12 (UTC+2) applications meant for a new internal development cluster were installed on other customer clusters, which were also modified.

The applications installed on the affected clusters could not run, because the image pull secrets they needed did not exist there, so no image could be pulled. The unscoped deployment looked normal to the team watching the rollout.

Symptom Control plane unresponsive Image pull failures

Root cause

Root cause Faulty deployment

Component CI system Container registry and its credentials

Contributing factor No configuration tests Excessive blast radius Missed in review

Trigger: A routine internal operation onboarded a new Managed Kubernetes development cluster. The scoping configuration file for the onboarding operator failed to reach the deployment branch because of a merge conflict, and at the same time a second change removed the operator from the exclusion list, which deployed it at once.

Detection

–

Remediation

Corrected the onboarding operator so it needs an explicit namespace configuration before acting on any cluster
Reconfigured the deployment system to fail deployments when required configuration files are missing
Rolled back the identified changes on the affected resources
Ran a recovery script, validated in staging, that removed all unintended installations and updates from the affected clusters

Action items

Add a mandatory CI/CD step that confirms all required configuration files reached the target branch before any deployment starts
Require additional peer review and change management approval for every deployment that touches production, whatever its size
Give teams a way to inform each affected customer in time about the status of post-incident cleanup

Lessons

IONOS treats the unintended change to cluster configurations as a failure of the controls that should keep internal operations from affecting customer environments.

Sources

https://status.ionos.cloud/incidents/4y4f2kgqhkgk · official-status · archived copy

Related cases

IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failedRailway · 2026
same root cause · shared CI system
IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database clusterWikimedia Foundation · 2025
same root cause · shared CI system
IL-0552 CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UICircleCI · 2025
same root cause · shared CI system
IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hoursKubernetes project · 2019
same root cause · shared CI system
IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own servicesKubernetes project · 2019
same root cause · shared CI system

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CI system, Container registry and its credentials among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0234
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0234/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.