IL-0541 Twingate · 2023-02-14networking

Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to them

Twingate was adding spot instance scaling to its Relay cluster infrastructure and applied the change through Terraform to every Relay cluster at once, starting at 01:00 UTC on 14 February 2023. The deployed Terraform configuration contained an error that removed a network tag the Relay cluster needed for its network firewall rules to be set correctly. Without the tag, the clusters in us-east4 and europe-west6 stayed discoverable to Clients and Connectors but could not be reached, which left those clients deadlocked while trying to attach.

Only these two clusters were affected because a configuration difference that depends on overall cluster size gave them a different outcome. The staging environment did not reproduce that sizing difference, so testing there did not show the problem. Twingate noticed the drop in connections at 01:40 UTC, disabled the pods in both clusters at 02:04 UTC so that connections moved to the next closest Relay clusters, where overall connection metrics looked normal, and at 03:30 UTC corrected the Terraform error and redeployed the two clusters, which ended the incident.

Severity: –
Scope: Client and Connector connections to the Relay clusters in us-east4 and europe-west6; after failover, some slowness for customers close to those clusters
Users affected: –
Region: us-east4, europe-west6
Time to detect: –
Time to resolve: –

Symptoms

At 01:40 UTC on 14 February 2023, Twingate saw the number of connections drop in two of its Relay clusters, the one in us-east4 (Ashburn, Virginia) and the one in europe-west6 (Zurich, Switzerland).

Clients and Connectors could still discover the two Relay clusters but could not reach them, and stayed stuck trying to attach to them. Twingate also contacted its cloud infrastructure provider proactively to rule out a regional problem on the provider's side. Once the pods in the two clusters were disabled, Clients and Connectors reconnected to the next closest Relay cluster, and customers close to the two clusters could see some slowness because their traffic now went to clusters further away.

Symptom Network unreachable

Root cause

Root cause Configuration change

Component Terraform VPC route tables and firewall rules

Contributing factor Unrepresentative staging No staged rollout

Trigger: At 01:00 UTC on 14 February, Twingate started a Terraform change to all of its Relay clusters worldwide to add spot instance scaling; the rollout took about 45 minutes.

Detection

At 01:40 UTC Twingate noticed a decrease in the number of connections in the two clusters and started investigating.

Remediation

Disabled the pods in the two affected Relay clusters at 02:04 UTC so Clients and Connectors reconnected to the next closest Relay cluster
Corrected the Terraform configuration error and redeployed the affected clusters at 03:30 UTC

Action items

Make the staging environment reflect the sizing and configuration differences between Relay clusters
Create Terraform plans automatically for every environment and configuration variation on feature development branches
Extend Relay health checks so a cluster is not discoverable to Connectors and Clients when its network firewall tag is missing
Research and introduce staggered Terraform rollouts for the Relay cluster infrastructure
Add multi-cluster connectivity to Connectors so they handle regional Relay cluster problems automatically

Lessons

–

Sources

https://status.twingate.com/incidents/5y3blrdzvw25 · official-status

Related cases

IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutesRailway · 2024
same root cause · shared VPC route tables and firewall rules, Terraform
IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023
same root cause · shared Terraform
IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1
same root cause · shared Terraform
IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project · 2023
same root cause · shared Terraform
IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021
same root cause · shared Terraform
IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster serviceDFDS · 2020
same root cause · shared Terraform
IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019
same root cause · shared Terraform
IL-0519 Apps on the cloud.gov PaaS became unavailable after an AWS security group change blocked their outbound trafficcloud.gov · 2016
same root cause · shared VPC route tables and firewall rules

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Terraform, VPC route tables and firewall rules among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0541
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0541/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.