Incident Library
Applied filters Root causeConfiguration changeFactorConfiguration not in codeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'
    MIT Open Learning2023
    Symptoms

    On 30 October 2023 at 3:51PM, six minutes after a configuration update was deployed to the xPro production Heroku stack, Pingdom alerted the DevOps on-call engineer, who confirmed at 3:52PM that the site was not responding.

    Service outageCredentials unavailable
    Trigger

    A configuration update applied through Salt at 3:45PM found that the Vault lease of the database credentials was about to expire and could not be renewed, so Vault generated new credentials.

    Root cause
    Configuration change
    Component
    DatabaseSecrets store
    Contributing factor
    Unreviewed defaultsChange with a delayed effectConfiguration not in code
    Detection

    Pingdom alerted the DevOps on-call engineer at 3:51PM, and the alert went through OpsGenie; Pingdom closed it automatically at 4:25PM.

    Remediation
    • Created a new database parameter group and applied it to the running RDS instance
    • Cleared the cached credentials and re-applied the Heroku configuration through Salt
    Action items
    • Move the management of xPro resources, including the RDS instance, into Pulumi, where AWS default parameter groups are not used
    Lessons

    Almost none of the configuration for this stack, the RDS instance in particular, was managed as code, and that was one of the main reasons the fix took long. Under Pulumi the instance would not have used an AWS default parameter group, and the fix would have been a change to an attached group.

    View case
  2. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022
    Symptoms

    From 14:43 to 15:03 UTC on 24 October 2022, during regular maintenance of the Kubernetes cluster that hosts PyPI, PyPI's backends could not be reached over HTTPS. The CDN could not fetch pages, uploads to uploads.pypi.org were impossible, and other services such as the legacy file redirect service were interrupted.

    Service outage
    Trigger

    Regular maintenance did a rolling restart of every node in the cluster to roll out upgrades and new configuration; as the new Kubernetes API server hosts came up, Kubernetes validated and refreshed the Ingress configurations.

    Root cause
    Configuration change
    Component
    Ingress controllerCloud load balancer
    Contributing factor
    Configuration not in codeChange with a delayed effect
    Remediation
    • Pointed the Ingress configuration at the new ACM TLS certificate, and Kubernetes restored the HTTPS listener on the ELB
    Action items
    • Investigate managing the hostnames on the ACM TLS certificates for PyPI's Ingress through Kubernetes resources instead of the AWS console
    Lessons

    When every resource is managed through the Kubernetes API, drift between the desired state and reality is less likely to build up and then surface during later maintenance.

    View case
Check your cluster for this

These 2 cases involve Cloud load balancer and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Cloud load balancer1Database1Ingress controller1Secrets store1
Contributing factor
Configuration not in code2Change with a delayed effect2Unreviewed defaults1
Organisation
MIT Open Learning1Python Software Foundation1
  1. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'
    MIT Open Learning2023
  2. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022