Incident Library

Organisation

Mailchimp

Mailchimp · mailchimp

1 case · 2019

Root cause Capacity exhaustion
Component PostgreSQL vacuumDatabase
Contributing factor Signal buried in noiseIgnored early signalMonitoring blind spotNo alert on limitRetry storm
Symptom Node disk fullThroughput degradationLoad imbalanceWrites rejected

Cases

  1. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hoursMailchimp · 2019

    Root cause Capacity exhaustion