Cases20 cases

16 to 20 of 20

1. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN

   bunny.net2021

   Symptoms

   On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

   Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance

   Trigger

   At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

   Root cause

   Faulty deployment

   Component

   Authoritative DNS servers and recordsCDN and edge networkObject storage

   Contributing factor

   Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action

   Detection

   About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

   Remediation

   • Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
   • Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
   • Migrated the deployment systems and files to a third-party cloud storage service
   • Rewired the deployment system and DNS software to the new storage and deployed
   • Synced the databases and deployed the newest file sets through the third-party service

   Action items

   • Phase out the BinaryPack library as a hot-fix
   • Test third-party libraries more extensively
   • Migrate the internal APIs to an independent third-party service
   • Find how to stop non-critical software from becoming a single point of failure across clusters
   • Make the DNS system keep a local copy of all backup data with automatic failure detection

   Lessons

   bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

   View case
2. IL-0398 GOV.UK showed "Error 503 Service Unavailable" pages for under an hour during a global outage of its primary CDN

   Government Digital Service2021

   Symptoms

   On 8 June 2021, for less than an hour, people visiting GOV.UK could not reach its information and services and got error pages with the heading "Error 503 Service Unavailable" instead. The outage was in the primary content delivery network (CDN) that GOV.UK uses, and many other websites on the same CDN around the world were down at the same time.

   Service outage

   Root cause

   Dependency failure

   Component

   CDN and edge network

   Contributing factor

   –

   Detection

   The incident was detected 4 minutes after it began, and an incident lead and a communications lead then started the formal incident process.

   Remediation

   • Waited 15 minutes to judge the primary CDN outage, then decided to fail over to the secondary CDN
   • Deployed the prepared DNS record changes, and traffic began to route to the secondary CDN within 30 minutes
   • Switched back to the primary CDN after it recovered, and monitored for an hour before declaring the incident resolved

   Action items

   • Hold a formal incident review of what happened and how the team responded
   • Consider how to improve the speed and transparency of communications
   • Consider how planned changes to platform technology and processes can make recovery from future incidents faster

   Lessons

   The CDN sits at the edge of the network, so when it breaks the whole website goes down, and the team keeps an always-on secondary CDN and a prepared DNS change for that case. Failing over too early can make an incident worse, because the secondary CDN gives a degraded service and primary CDN incidents are often resolved quickly, so the plan waits 15 minutes before failing over.

   View case
3. IL-0491 Squarespace's CDN origin fleet fell below its healthy node threshold, and images, scripts and stylesheets failed to load

   Squarespace2019

   Symptoms

   At 12:10 AM ET on Tuesday, February 12, 2019, Squarespace's monitoring alerted the on-call engineer for the CDN origin fleet that the fleet had fallen below a safe number of healthy nodes. Requests to static.squarespace.com and static1.squarespace.com failed, so images, scripts and style sheets did not load for some sites, and the status page later reported elevated error rates serving media and stylesheets.

   High error rate

   Trigger

   At 10:30 PM ET on February 11, an engineer started a maintenance operation across the CDN origin fleet. The operation kept running in the background after the engineer believed it had been terminated.

   Root cause

   Operator error

   Component

   CDN and edge network

   Contributing factor

   Excessive blast radiusMonitoring blind spot

   Detection

   The monitoring system alerted the CDN origin fleet's on-call engineer at 12:10 AM ET when the fleet fell below a safe number of healthy nodes.

   Remediation

   • Restored the CDN origin fleet, partially by 2:15 AM ET and to normal behaviour at 2:20 AM ET

   Action items

   • Revise the incident-response playbooks with checks and actions for this fleet, starting with an assessment of customer impact and the immediate involvement of an Incident Commander
   • Improve external monitoring to assess customer impact of issues with the CDN partners and origin services more quickly
   • Revise operational policies and add safeguards for issuing maintenance commands, including limiting the share of the fleet a command can operate on
   • Reduce the need for direct manual maintenance operations
   • Create independent failure domains and route requests automatically around unhealthy parts of the system

   View case
4. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests

   Google Cloud2018

   Symptoms

   From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.

   High error rateConnection resetsHigh latency

   Trigger

   A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.

   Root cause

   Software defect

   Component

   Cloud load balancerCDN and edge network

   Contributing factor

   Gap in automated testsExcessive blast radius

   Detection

   Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.

   Remediation

   • Reverted the configuration change at 12:44 PDT, which stopped the GFE restarts

   Action items

   • Add safeguards that disable features not yet in service
   • Harden the GFE testing stack against latent bugs in production binaries that make a task restart
   • Isolate the shards of GFE pools further from each other to reduce the scope of failures
   • Build a consolidated dashboard of all configuration changes for GFE pools

   View case
5. IL-0388 A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guilds

   Discord2017

   Symptoms

   At about 14:01 PDT on 13 October 2017, Discord engineers were alerted to anomalies by real-time metrics and monitoring. A Redis instance that was the primary of a highly available cluster used by the API services had dropped offline after Google Cloud Platform migrated it automatically, and some API instances did not handle the failover to the secondary node correctly.

   Failover failureHigh latencyService outage

   Trigger

   Google Cloud Platform automatically migrated the Redis instance that was the primary of a highly available cluster used by the API services, and the node incorrectly dropped offline.

   Root cause

   Software defect

   Component

   In-memory cacheDatabaseCDN and edge network

   Contributing factor

   Unpatched known bugOverlapping failuresMonitoring blind spot

   Detection

   Real-time metrics and monitoring alerted engineers at 14:01 PDT. The second phase was seen at 15:41 as another anomaly in monitoring, and end user reports of servers appearing offline were escalated at 15:42.

   Remediation

   • Ran a rolling restart of all API instances to clear the Redis connection issues
   • Corrected the misconfigured edge caching rule for the expensive API route
   • Disabled the problem API endpoint on the edge network
   • Recovered the Cassandra cluster
   • Ran a second rolling restart of all API instances for the latency left by the Cassandra issues
   • Removed the misbehaving guilds node from the service hashring
   • Forcefully restarted the misbehaving guilds node
   • Restarted the whole service, forcing all users to reconnect
   • Blocked traffic that relied on the failing push component to reduce its load
   • Corrected the API setting that lets users reconnect quickly and safely

   Action items

   • Fix the known issues with Redis failover within the coming weeks
   • Change the cached route so this type of error does not degrade other services
   • Monitor the signals that would have warned of the cascading failure
   • Improve the reliability and failure handling of the affected services
   • Roll out an update of the library with the suspected flaws

   View case

By root cause8