Cases2 cases

1 to 2 of 2

1. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case
2. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable

   2i2c2022

   Symptoms

   On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

   Service outageNode NotReady

   Trigger

   Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system image

   Contributing factor

   Limited debug access

   Detection

   The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

   Remediation

   • Started an upgrade of the cluster's Kubernetes and node versions, which replaced all old nodes with new working ones

   Action items

   • Document how to SSH into the Kubernetes nodes
   • Find a way to be notified of Azure advisories

   Lessons

   Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

   View case

By root cause1