Incident Library

IL-0326 MailPace · 2023-08-11database

A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hours

MailPace's clean-up cron job, which removes data older than each organisation's retention period, had been failing silently, so old data piled up. Under a normal daily load the task would have succeeded, but each destroy_all call removes all related records in more than 10 tables holding responses, attachments and other data, and accounts send thousands of emails a day, so a single call could be very large. Given 8gb of memory the Ruby code ran, and the database filled quickly; MailPace assumes the space went to Write Ahead Logs and was waiting for the provider to confirm. The database plan had also been sized without enough headroom for growth, and an upgrade was already planned.

At 13:37 MailPace contacted its managed database provider, and at 13:45 it began its tested restoration procedure, preparing an offsite backup from six hours earlier on a new database instance, which would have meant some data loss. At 14:30, just before the switch, the provider reported that it had been automatically notified at 99% storage and was moving the database to a larger instance; two of its support engineers working on the issue at the same time, one from the automated check and one from MailPace's ticket, slowed the migration. The database was read-only from 15:25 and fully back on the larger instance at 15:50. Afterwards MailPace replicated the problem on a separate instance restored from backup, removed the old data in production day by day with a revised query while watching memory and database size, and changed the job to delete in batches of 50 records four times a day.

Severity
–
Scope
All database transactions failed across MailPace's services and emails were not sent; during the migration users could log in but not send emails
Users affected
–
Region
–
Time to detect
–
Time to resolve
2h 14m

Symptoms

At 13:36 BST on Friday August 11th 2023, minutes after MailPace reran its data clean-up job with more memory, notifications arrived that all database transactions were failing across multiple services. The managed Postgres database had gone offline, did not come back when restarted, and reported that it was out of space.

The outage was preceded by signs found in routine checks at 13:00: the database that stores sent emails and API details was about 1.5x larger than it should have been and at about 80% capacity, and it held data older than the 30-day retention policy. The logs showed that the daily clean-up cron job, a Rake task that deletes each domain's old outgoing emails with destroy_all, had not been completing and was exiting with the message killed. Taking killed to mean the job ran out of memory, the team raised the job's memory from 1gb to 8gb at 13:15; the job then ran for a few minutes and the database filled.

The team could not see how a database it believed to be at 70% capacity could run out of space while data was being deleted. MailPace's status pages did not show the outage, presumably because https://app.mailpace.com still returned a 200 while the database was down. At 15:25, while the provider migrated the database, it was put in read-only mode: users could log in but not send emails, and the team at first thought it had fully recovered, then realised its mistake and updated Twitter. Emails went out again after full recovery at 15:50.

Symptom Node disk fullService outageWrites rejected

Root cause

Root cause Capacity exhaustion

Component Database
Contributing factor Inefficient queryMonitoring blind spotNo headroom
Trigger
The daily retention clean-up job, which had been failing with killed, was rerun with its memory raised from 1gb to 8gb, and its large destroy_all deletions filled the database disk.

Detection

Notifications that all database transactions were failing across multiple services arrived as soon as the outage began; the provider was automatically notified at 99% storage. MailPace's two status pages did not detect the outage.

Remediation

Action items

Lessons

Several causes came together: a clean-up job that failed silently, a naive deletion that grew huge once data had piled up, a database plan without headroom for growth, and status pages that missed the outage, presumably because https://app.mailpace.com still answered 200 while the database was down. MailPace also advises that critical services such as email sending be set up with more than one provider.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.