Incident Library
Applied filters Root causeFaulty deploymentFactorNo configuration testsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters
    IONOS Cloud2026
    Symptoms

    On 8 May 2026 IONOS Cloud reported "K8s Control Planes not available": some Managed Kubernetes customers in DE/TXL, DE/FRA and DE/FRA/2 had trouble connecting to the control plane and saw degraded Kubernetes functionality. Between 15:22 and 16:12 (UTC+2) applications meant for a new internal development cluster were installed on other customer clusters, which were also modified.

    Control plane unresponsiveImage pull failures
    Trigger

    A routine internal operation onboarded a new Managed Kubernetes development cluster. The scoping configuration file for the onboarding operator failed to reach the deployment branch because of a merge conflict, and at the same time a second change removed the operator from the exclusion list, which deployed it at once.

    Root cause
    Faulty deployment
    Component
    CI systemContainer registry and its credentials
    Contributing factor
    No configuration testsExcessive blast radiusMissed in review
    Remediation
    • Corrected the onboarding operator so it needs an explicit namespace configuration before acting on any cluster
    • Reconfigured the deployment system to fail deployments when required configuration files are missing
    • Rolled back the identified changes on the affected resources
    • Ran a recovery script, validated in staging, that removed all unintended installations and updates from the affected clusters
    Action items
    • Add a mandatory CI/CD step that confirms all required configuration files reached the target branch before any deployment starts
    • Require additional peer review and change management approval for every deployment that touches production, whatever its size
    • Give teams a way to inform each affected customer in time about the status of post-incident cleanup
    Lessons

    IONOS treats the unintended change to cluster configurations as a failure of the controls that should keep internal operations from affecting customer environments.

    View case
  2. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025
    Symptoms

    At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

    Service outageLock contention or deadlockDatabase overload
    Trigger

    A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    No configuration testsLimited debug access
    Detection

    Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

    Remediation
    • The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered
    Action items
    • Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
    • Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
    • Configure database user connection limits so administrative access stays possible during incidents
    View case
  3. IL-0554 A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutes
    Sentry2024
    Symptoms

    From 3:42 PM to 4:04 PM PT on 10 October 2024, the Sentry US site and event ingestion were unavailable, and the status page reported an issue with the Postgres database that made the site unavailable. The dashboard, the API and every US ingestion type, including errors, transactions, attachments, profiles, replays, spans, cron monitoring and custom metrics, were affected.

    Service outageDatabase overload
    Trigger

    A database migration ran certain database operations through the RunSQL command without lock and statement timeouts.

    Root cause
    Faulty deployment
    Component
    DatabaseSchema migration
    Contributing factor
    No configuration testsNo runbook
    Remediation
    • Issued a fix, after which the dashboard and ingestion recovered
    Action items
    • Replace RunSQL commands with more robust solutions in the migration framework
    • Update the database framework to the latest version, which handles some operations RunSQL was used for
    • Write runbooks for recovering from lock or timeout issues during migrations
    • Improve alerting and observability so the team reacts sooner
    View case
Check your cluster for this

These 3 cases involve Database and CI system. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment3 cases

Component
Database2CI system1Database proxy and connection pooler1Container registry and its credentials1Schema migration1
Contributing factor
No configuration tests3Excessive blast radius1Limited debug access1No runbook1Missed in review1
Organisation
IONOS Cloud1Railway1Sentry1
  1. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters
    IONOS Cloud2026
  2. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025
  3. IL-0554 A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutes
    Sentry2024