Incident Library

IL-0511 MIT Open Learning · 2023-10-30databaseaws

MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'

At an undetermined point in the six months before the outage, the RDS instance behind xPro production started to create new credentials with SCRAM password authentication instead of MD5 by default. The pgbouncer buildpack the team used on Heroku did not support SCRAM. Database credentials for the Heroku applications were generated by Vault through SaltStack and cached on the Salt master.

On 30 October 2023 at 3:45PM a configuration update applied through Salt saw that the Vault lease was due to expire and could not be renewed, and Vault generated new credentials with the new SCRAM default. pgbouncer sent an MD5 password hash, RDS rejected it, and since the application cannot start without a database connection, xPro stopped serving requests. Changing the Vault role did not help. The team then created a new database parameter group, applied it to the running database at about 4:20PM, re-applied the Heroku configuration with cleared cached credentials, and confirmed at 4:25PM that the site, logins and other database operations worked. The RDS instance had used an AWS default parameter group, which cannot be modified, so the group had to be duplicated, changed and attached, which took longer than editing an attached group would have.

Severity
–
Scope
The xPro production site was not responding
Users affected
–
Region
–
Time to detect
6m
Time to resolve
40m

Symptoms

On 30 October 2023 at 3:51PM, six minutes after a configuration update was deployed to the xPro production Heroku stack, Pingdom alerted the DevOps on-call engineer, who confirmed at 3:52PM that the site was not responding.

The Heroku logs showed pgbouncer failing to log in to the database with a user generated by Vault: "cannot do SCRAM authentication: wrong password type", followed by "closing because: server login failed: wrong password type (age=14s)". The team recognised the failure from an earlier one seen while setting up the new MITOpen environments and found the pull request that had fixed it there.

That fix could not be applied quickly, because the configuration of this environment was not managed by Pulumi and its code could not be found. The team first changed the role definition in Vault and re-applied the Heroku configuration twice with cleared cached credentials. The new credentials were confirmed to be different, but the site stayed down.

Symptom Service outageCredentials unavailable

Root cause

Root cause Configuration change

Component DatabaseSecrets store
Contributing factor Unreviewed defaultsChange with a delayed effectConfiguration not in code
Trigger
A configuration update applied through Salt at 3:45PM found that the Vault lease of the database credentials was about to expire and could not be renewed, so Vault generated new credentials.

Detection

Pingdom alerted the DevOps on-call engineer at 3:51PM, and the alert went through OpsGenie; Pingdom closed it automatically at 4:25PM.

Remediation

Action items

Lessons

Almost none of the configuration for this stack, the RDS instance in particular, was managed as code, and that was one of the main reasons the fix took long. Under Pulumi the instance would not have used an AWS default parameter group, and the fix would have been a change to an attached group.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Secrets store among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.