Incident Library

IL-0523 U.S. Department of Health and Human Services · 2023-09-07database

TANF-app production deployments were blocked for two months when a search-index migration ran out of database disk space

TANF-app is a Django application whose production database is a PostgreSQL instance on cloud.gov. The team counts the incident, from 7 September to 9 November 2023, as several incidents combined. A previously deployed search_indexes migration, 0016, had been deleted and later recreated, which caused the first deployment failures and made fixing deployments more complex. When the sprint 82 release ran on 20 September, the large migration search_indexes 0017 grew the database's temporary space beyond the instance storage and failed with a disk space error, and production stayed blocked for two months while staging problems and blocked pipelines added delay.

In November the team reproduced the disk failure in its raft environment by lowering the instance storage to 20gb, and listed ways out: split large migrations into smaller ones, raise the postgres tmp_file_limit, increase the instance storage in cloud.gov, or wipe the temp table space by creating a new instance from a backup. On 8 November the production instance size was increased and the migrations were re-run, which exposed a second problem: the migration failed converting a blank string to an integer. T3 rows submitted with a single case and no space-filling for the second case had passed through the value_is_empty check during parsing because of a Python substring quirk and were stored as blank strings. Correct field types in the models would have kept the migrations from failing.

On 9 November, to avoid creating a new database, the team backed up the production database, truncated all of the search_indexes tables, whose data it was not worried about losing, and redeployed the sprint 82 release. Production was restored to sprint 82 and deployments could move forward. The blank-string parsing bug was fixed in a separate pull request.

Severity
–
Scope
TANF-app production deployments were blocked from 7 September to 9 November 2023
Users affected
–
Region
–
Time to detect
–
Time to resolve
63d

Symptoms

On 20 September 2023 the sprint 82 production deployment of TANF-app failed with a disk space error while it ran a large search-index migration, search_indexes 0017, against the production PostgreSQL database on cloud.gov. Production could not be deployed again until 9 November.

The deployment trouble had begun on 7 September with the failed sprint 80 deployment. A search_indexes migration that had already been deployed, 0016, was deleted, and makemigrations and migrate were commented out in the deployed environments to force manual migrations. The sprint 82 release uncommented them and recreated the broken 0016. Between the two deployments, develop pipelines were blocked for various reasons, and after 20 September staging problems kept the release from reaching production.

On 1 November the team tried to reproduce the failure in its raft environment with 700k to 1m rows each of t1, t2, t3, t4 and t5. During the migrations the database size grew up to 8x and fell back to its starting level when they finished, and temp_files grew by about 4gb after completion, but the migrations ran even with far more records than production. The team put this down to a difference in some database settings. On 6 November, lowering the raft database instance storage to 20gb reproduced the failure after several migration runs, as the temp space eventually exceeded the instance size.

On 8 November the production database instance size was increased and the migrations were run again. They then failed with a new error about converting a blank string to an integer. Every character field in T3 had some records with the value '' (blank string) instead of None.

Symptom Deploys failingNode disk full

Root cause

Root cause Faulty deployment

Component DatabaseSchema migration
Contributing factor Overlapping failuresGap in automated testsUnrepresentative stagingNo runbook
Trigger
The sprint 82 deployment on 20 September 2023 recreated the deleted search_indexes migration 0016 and ran the large migration search_indexes 0017 against the production database.

Detection

The sprint 82 production deployment failed with a disk space error, and the large migration search_indexes 0017 was identified as the cause.

Remediation

Action items

Lessons

The work was unplanned and pulled in many developers from a small team, so the team wants documentation and playbooks that let fewer people step away from current work. Large datasets and unexpected edge cases such as rows without space-fill had not been tested, and a copy of the production database with PII scrubbed would allow tests against real-world data. Deploying in-progress code to other environments and changing migrations afterwards had caused conflicts there.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Schema migration among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.