Incident Library

Organisation

MIT Open Learning

MIT Open Learning · mit-open-learning

3 cases · 2023 to 2026

Root cause Configuration change2Faulty deployment1
Component Database2CDN and edge network1cert-manager1Authoritative DNS servers and records1Secrets store1
Contributing factor Change with a delayed effect2Configuration not in code1Unreviewed defaults1Inefficient query1Harmful manual action1Monitoring blind spot1No configuration tests1Unrepresentative staging1
Symptom Service outage3Credentials unavailable1Database overload1Lock contention or deadlock1Node disk full1TLS certificate errors1

Cases

  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedMIT Open Learning · 2026 · sev1

    Root cause Configuration change

  2. IL-0510 MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hoursMIT Open Learning · 2026

    Root cause Faulty deployment

  3. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'MIT Open Learning · 2023

    Root cause Configuration change