Graph
Cases6 cases
1 to 5 of 6
- Symptoms
From 16:18 UTC on 28 May 2026 the Lucidworks AI hosted LLMs in the us-southcarolina region were unavailable: queries to llama-3-8b-instruct, llama-3v2-3b-instruct and phi-4-multimodal-instruct returned 500 or 429 errors. Search, embedding models, the Lucidworks Platform UI and passthrough LLM calls kept working.
- Trigger
A routine Kubernetes patch upgrade on a cluster in us-southcarolina cycled its node pools one after another in quick succession.
- Root cause
- Component
- Contributing factor
- Remediation
- Deleted the degraded replacement head node that could not pull container images
- Waited for the LLM image pulls to finish on the replacement nodes
- Launched new nodes with the required LLM images and redirected Lucidworks AI traffic to them
- Action items
- Upgrade clusters that host LLM workloads in sequence, with each node pool fully healthy before the next one is upgraded
- Investigate image pre-loading, such as pre-baked disks or image streaming, for large ML model images
- Open a support ticket with the cloud provider about the abnormal Docker pull times
- Coordinate Kubernetes maintenance windows with LLM service owners so they do not clash with ongoing deployments
- Build tooling to force a Ray cluster failover to the blue or green state instead of waiting for Ray to settle it
IL-0552CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UICircleCI2025- Symptoms
At about 23:00 UTC on 21 January 2025, an automated alert reported that the CircleCI database instance holding archived data was almost out of free storage space. From 23:50 UTC customers saw slow starts and cancellations of workflows and jobs, and delays and difficulty viewing workflows in the CircleCI UI, although pipelines for a specific project could still be viewed.
- Trigger
A code change to a function in the archival database on 16 January 2025 unintentionally made it write an excessive number of log messages, and the database filled its storage.
- Root cause
- Component
- Contributing factor
- Detection
An automated alert at about 23:00 UTC warned that the archival database was almost out of free storage, before customers were affected. The growing workflows queue was not linked to it at first because a separate alert about request volume fired at about the same time.
- Remediation
- Scaled up the event consumer that processes the workflows queue at 23:44 UTC
- Promoted a read replica of the archival database to a standalone primary at 23:55 UTC
- Action items
- Fix the database function so it no longer writes an excessive number of log messages
- Add a max duration to the workflows retry policy for archiving so it fails earlier than the default timeout
- Move workflow archival to an event-based model that decouples it from the workflows service
- Add forecast and anomaly monitoring for databases that alerts on unusual activity before it reaches critical levels
- Lessons
The storage alert on the archival database did fire, but the database filled much faster than forecast, so there was little time to act before customers were affected. Because the workflows service called the archival service synchronously, a problem in the archival database slowed the whole workflows service.
- Symptoms
On 20 September 2023 the sprint 82 production deployment of TANF-app failed with a disk space error while it ran a large search-index migration, search_indexes 0017, against the production PostgreSQL database on cloud.gov. Production could not be deployed again until 9 November.
- Trigger
The sprint 82 deployment on 20 September 2023 recreated the deleted search_indexes migration 0016 and ran the large migration search_indexes 0017 against the production database.
- Root cause
- Component
- Contributing factor
- Detection
The sprint 82 production deployment failed with a disk space error, and the large migration search_indexes 0017 was identified as the cause.
- Remediation
- Increased the production database instance size and re-ran the migrations
- Created a backup of the production database
- Truncated all of the search_indexes tables
- Redeployed the sprint 82 release
- Fixed the bug that stored empty fields as blank strings
- Action items
- Write troubleshooting steps for database problems into the technical documentation
- Write wiki pages and operations playbooks that outline common problems and responses
- Add a database section and the cf revisions command to the failed-deployment troubleshooting document
- Keep a GitHub repository of changes made directly to production
- Save terminal output from production debugging sessions
- Create Django migration best practices to enforce in code reviews
- Do not edit migrations that have been merged to develop; write a new migration instead
- Rename auto-generated migrations to more descriptive names
- Use one auto-generated migration per model change and regenerate when a pull request makes successive changes
- Organise a discussion on database seeding and backups for testing
- Lessons
The work was unplanned and pulled in many developers from a small team, so the team wants documentation and playbooks that let fewer people step away from current work. Large datasets and unexpected edge cases such as rows without space-fill had not been tested, and a copy of the production database with PII scrubbed would allow tests against real-world data. Deploying in-progress code to other environments and changing migrations afterwards had caused conflicts there.
IL-0591Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster releaseGitpod2022- Symptoms
From April 11, 2022 at 13:17 UTC customers and Gitpod staff reported that Gitpod workspaces were intermittently becoming unresponsive. CPU usage of nodes and pods looked normal, with nothing maxed out, and at 13:58 UTC the team found that the workspaces were IO bound rather than CPU bound.
- Trigger
The ws39 generation of workspace clusters, released to all regional clusters on April 8th at 20:08 UTC and to the XL clusters on April 9th at 17:45 UTC, split the 1TB disk into smaller RAID0 disks, changed CPU limiting and workspace networking, and moved to faster machine types.
- Root cause
- Component
- Contributing factor
- Detection
Customers and Gitpod staff reported on April 11th at 13:17 UTC that workspaces were intermittently unresponsive, and the team opened an incident in Slack through its incident.io integration. On April 17th the team noticed increased CPU load on the workspace nodes and opened an internal incident.
- Remediation
- Shifted workspace starts back to the ws38 clusters on April 11th
- Built ws39a clusters with c2d machines and local SSDs and moved traffic to them
- Requested a limit increase from Let's Encrypt
- Routed EU XL traffic to the regional EU cluster and built ws-eu39bxl to replace the crashed ws-eu39xl
- Cleaned up workspaces stuck in a stopping state so users could start them again
- Released ws40 with 1TB boot disks to get more IO bandwidth
- Removed the workspaces that consumed excessive CPU time for crypto-mining
- Blocked the fraudulent users behind the crypto-mining from using Gitpod
- Rolled back the ws-daemon change that broke CPU limiting and shipped the fix in the ws34 clusters
- Blocked ACME HTTP requests to workspace ports so Let's Encrypt rate limits would not be hit again
- Rolled out IO limiting with the ws41xl and ws41 clusters and moved traffic to them step by step
- Action items
- Invest in test automation so that test results stop releases like this one
- Vary the interval between traffic shifts to new clusters, with more time for larger releases
- Add a risk assessment stage, to be automated, that governs changeset size and rollout speed of workspace cluster rollouts
- Reduce the impact teams have on each other
- Add review criteria that find backwards incompatible changes before pull requests are merged, and split such pull requests
- Recommend rollback as the primary means of mitigation in the incident procedures, and look at automating it
- Make the status page show the workspace service as down when it is, and update the playbooks
- Keep incident communication in a single Slack channel and Google Meet
- Put 10% of the engineering staff on the on-call rotation from May 9th and use that time on observability, alerting and stability
- Lessons
Gitpod concluded that it should have committed to rolling back on April 11th instead of building and shipping fixes, which kept customers on degraded workspaces longer than necessary. A large release so soon after the previous one needed a risk assessment and slower traffic shifts, and backwards incompatible changes that block a rollback need to be found before pull requests are merged. Failure-free operations require experience with failure.
IL-0566Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenanceDevsisters2021- Symptoms
About half an hour after the Cookie Run: Kingdom guild update maintenance ended at 11:00 on 6 March 2021, one CockroachDB node was using all 24 cores of its 24-core machine and server metrics began to show a significant number of errors. Soon two more nodes showed abnormal CPU use.
- Trigger
During the guild update maintenance, new guild tables were added and pre-split across the cluster, and a new database node was added to replace one removed a few days earlier for a physical hardware issue. When the game reopened, all players at once wrote to existing tables whose primary-key prefixes were new or had changed in the update.
- Root cause
- Component
- Contributing factor
- Detection
Monitoring after the maintenance showed one node's CPU at the ceiling together with errors in server metrics. Later a Slack alert reported a node failing its Host Status Check, and the abnormal-log channel reported filemap_fdatawait_range_keep_errors.
- Remediation
- Blocked new logins without kicking players already in the game, using the maintenance switch as a makeshift access queue
- Put the game into full emergency maintenance at 14:50 when several nodes failed health checks
- Backed up the game data table by table after the full database backup stalled
- Paused writing play records to reduce load on the database before reopening
- Split the problem tables from about 18:30
- Decommissioned the node that failed its Host Status Check
- Reopened the game and used the login-block queue again when a node reached 100% CPU
- Terminated the node that kept returning to 100% CPU
- Lessons
In a single-server relational database a change to the primary-key prefix is usually harmless, but in CockroachDB's distributed design such a change on a production cluster needs great care. The team had studied this in CockroachDB documentation, but living through it taught far more than reading about it.
These 6 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause1
Faulty deployment6 cases
- Component
- Contributing factor
IL-0237A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went downLucidworks2026IL-0552CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UICircleCI2025IL-0523TANF-app production deployments were blocked for two months when a search-index migration ran out of database disk spaceIL-0591Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster releaseGitpod2022IL-0566Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenanceDevsisters2021IL-0278A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failedRedis2016