Cases6 cases

5 to 6 of 6

1. IL-0446 About 20% of requests to an ITANDI ECS service got ALB 5XX errors, apparently from one zone, right after a deploy

   ITANDI2020

   Symptoms

   Right after a deploy to the production ECS (Fargate) service, HTTPCode_ELB_5XX_Count on the service's ALB jumped. The service receives about 10,000 requests a minute, and roughly 20% of them got a 5XX.

   High error rate

   Trigger

   A deploy recreated the ECS tasks, and at that moment communication between the ALB and its backends became unstable.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerAmazon ECS

   Contributing factor

   –

   Detection

   The team saw HTTPCode_ELB_5XX_Count jump on the ALB right after the deploy, together with target connection errors.

   Remediation

   • Restarted the ECS tasks, which did not help
   • Moved the ALB's ap-northeast-1d subnet to another subnet in the same zone, which did not help
   • Moved the six ECS tasks running in 1d to 1a and 1c, which did not help
   • Removed the 1d subnet from the ALB, which did not help
   • Changed the ALB idle timeout, which did not help
   • Rolled the ECS tasks back to the version before the deploy, which did not help
   • Created a new ALB with the same VPC, subnets, backends and security groups
   • Attached both ALBs to the ECS service
   • Sent one tenth of the traffic to the new ALB with a weighted DNS record
   • Moved all traffic to the new ALB after it returned no 5XX

   Lessons

   When a graph shows something that should be impossible on the network, suspect the AWS side as well.

   View case
2. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung

   Netlify2016

   Symptoms

   At 8am PST on Monday, August 22, 2016, Netlify got alerts that its central load balancer, the single public IP that users point A records at, had stopped responding. Rackspace had null-routed the load balancer, so all traffic to that IP hung before it reached Netlify's servers.

   Network unreachable

   Trigger

   A DDoS attack against Netlify's central load balancer in Rackspace's Chicago datacenter, after which Rackspace null-routed the load balancer's public IP.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancer

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts at 8am PST reported that the central load balancer had stopped responding, and Rackspace then confirmed it had null-routed the load balancer.

   Remediation

   • Started a new cluster of CDN machines in Google Compute behind a Google network load balancer
   • Gave clients who had contacted support the new public IP and told them to change their A records
   • Updated the documentation and announced the new IP on Twitter, Gitter and the status page

   Action items

   • Keep the new Google-based network IP running in place of the Rackspace load balancer
   • Release Netlify's own DNS hosting, integrated with the CDN, so domains do not depend on a fixed IP
   • Fix the status page publishing process so it is updated straight away during an issue
   • Build a mailer that sends targeted mails to all paying clients during an outage

   Lessons

   Netlify judged Rackspace's load balancer too vulnerable to DDoS and Rackspace's DDoS processes not up to the standard of other cloud providers, and expected Google's anycast-based Maglev load balancer to be more resilient. In the end, the only way to protect a domain from DDoS-related outages is not to expose any fixed IP address for it, which is why sites should use the www domain with a CNAME, or Netlify DNS, rather than a single A-record IP.

   View case

By root cause1