Incident Library

IL-0446 ITANDI · 2020-01-28 (published)awsnetworking

About 20% of requests to an ITANDI ECS service got ALB 5XX errors, apparently from one zone, right after a deploy

The service runs on AWS ECS (Fargate) and is deployed with CircleCI and Terraform: merging a pull request makes Terraform recreate the ECS tasks with the new container image. Right after one such deploy, about 20% of requests got 5XX from the ALB, apparently only through ap-northeast-1d. Restarting tasks, changing the ALB's subnet in 1d, moving tasks out of 1d, removing 1d from the ALB, changing the idle timeout and rolling back the deploy all failed to stop the errors.

The team then rebuilt the load balancer without switching all traffic at once. It created a new ALB with exactly the same VPC, subnets, backends and security groups, attached it to the same ECS service (one ECS service can have several load balancer target groups), and added a second weighted DNS record in the hosted zone pointing at the new ALB with one tenth of the traffic. By about 4 a.m. the new ALB had returned no ELB 5XX at all, so the weights were set to 10 for the new ALB and 0 for the old one. The old ALB was kept and the team asked AWS Support to investigate. AWS Support answered that a load balancer node had developed a fault at the time of the deploy, which caused the errors.

Severity
–
Scope
About 20% of the roughly 10,000 requests a minute to one service got 5XX responses from the ALB
Users affected
–
Region
ap-northeast-1 (errors appeared through ap-northeast-1d)
Time to detect
–
Time to resolve
–

Symptoms

Right after a deploy to the production ECS (Fargate) service, HTTPCode_ELB_5XX_Count on the service's ALB jumped. The service receives about 10,000 requests a minute, and roughly 20% of them got a 5XX.

Target connection errors appeared at the same time. Restarting the ECS tasks did not bring the service back.

In CloudWatch, the ELB 5XX responses appeared to come only from traffic through ap-northeast-1d, one of the three availability zones of the ALB. AWS documentation lists problems in the backends as the cause of ELB 5XX, but an error in only one zone of three looked impossible unless created on purpose. Checks of the subnet ACLs, the ECS security groups and CloudTrail found nothing wrong.

Assuming a network problem on the AWS side in 1d, the team moved the ALB's 1d subnet to another subnet in 1d (dig confirmed the ALB name now resolved to different IPs), moved the 6 of 15 ECS tasks running in 1d to 1a and 1c, and removed the 1d subnet from the ALB. None of this helped. Assuming the deploy was at fault, the team changed the ALB idle timeout in case slower tasks were timing out, and then rolled the ECS tasks back to the version before the deploy. The errors continued; by then it was about 3 a.m.

Symptom High error rate

Root cause

Root cause Cloud provider outage

Component Cloud load balancerAmazon ECS
Contributing factor –
Trigger
A deploy recreated the ECS tasks, and at that moment communication between the ALB and its backends became unstable.

Detection

The team saw HTTPCode_ELB_5XX_Count jump on the ALB right after the deploy, together with target connection errors.

Remediation

Action items

–

Lessons

When a graph shows something that should be impossible on the network, suspect the AWS side as well.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer, Amazon ECS among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.