Incident Library

IL-0387 Mailgun · 2017-01-12database

Mailgun MongoDB secondaries were overloaded by rising connections, and a rollback then moved reads onto the primary too

The MongoDB secondary servers that handle most of Mailgun's reads in one production region became overloaded as connections to them rose gradually, which also caused the elevated errors in the click and open tracking services. Because the rise coincided with MongoDB configuration changes made the previous day, the team started rolling those changes back at 15:15 UTC through several new pull requests. Through human error, the rollback also moved the API and SMTP services from reading on the secondaries to the primary, and the MongoDB primary could not handle the load, so from 15:55 UTC the API, SMTP and website returned more errors.

Between 16:00 and 19:00 UTC the team resized the MongoDB servers and added server-level connection limits, which brought the API back to about 75% of normal throughput. At 20:28 UTC the team began deploying limits on how many connections each service could open to the primary, together with a redeploy of the configuration that prefers secondaries for reads, and services returned to normal at 20:35 UTC. The MongoDB logs reviewed afterwards showed a broad loss of cluster performance but no set of queries that explained the rise in connections. The tracking services had run from a single region until then.

Severity
–
Scope
Click and open tracking services, then the API, SMTP and website in one production region; the API served about 75% of typical throughput between 16:00 and 19:00 UTC
Users affected
–
Region
–
Time to detect
–
Time to resolve
6h 26m

Symptoms

On 12 January 2017 at 14:09 UTC, Mailgun engineers began responding to alerts for the click and open tracking services and the primary MongoDB clusters in one production region. Within ten minutes the engineers found that the MongoDB secondary servers, which serve most reads, were under heavy load and slow to answer, which also caused the tracking errors. The status page reported intermittent timeouts on the Control Panel, API, SMTP and tracking links.

Metrics showed a gradual rise in connections to the secondary servers. From 15:55 UTC, after a rollback, the API, SMTP and website had elevated error rates because the MongoDB primary could not keep up with the requests and became unstable. Between 16:00 and 19:00 UTC the API in this region served about 75% of its typical request throughput.

Symptom Database overloadIntermittent timeoutsHigh error rate

Root cause

Root cause unknown

Component Database
Contributing factor Harmful manual actionSingle point of failure
Trigger
A gradual increase in connections to the MongoDB secondary servers, which coincided with several MongoDB configuration changes made the day before. A rollback of those changes then also introduced, by mistake, a change that moved the API and SMTP services from the secondaries to the primary.

Detection

Alerts for the click and open tracking services and the primary MongoDB clusters at 14:09 UTC.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.