Incident Library
Applied filters SymptomDeploys failingFactorMonitoring blind spotOrganisationBinder ProjectClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use
    Binder Project2022
    Symptoms

    On 2 June 2022 from 10:00 CET, mybinder.org stopped launching any new pods, and nobody on the team noticed until a user reported a Binder outage in the Matrix channel at 21:00.

    Deploys failingService outage
    Root cause
    Software defect
    Component
    GKE control plane
    Contributing factor
    Monitoring blind spot
    Detection

    A user reported a Binder outage in the Matrix channel at 21:00, and a team member who saw it found that pods had not been launching for several hours.

    Remediation
    • Deleted the gke-resource-quotas ResourceQuota, after which pods could be created again
    Action items
    • Address uptime monitoring and alerting
    Lessons

    It took 11 hours to realise there was a major outage. Once the right object was known, deleting it fixed the problem quickly, and the team was lucky that someone with the skills and permissions to make the change was at their computer at 11pm their time.

    View case
Check your cluster for this

This case involves GKE control plane. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
GKE control plane
Contributing factor
Monitoring blind spot
Organisation
Binder Project
  1. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use
    Binder Project2022