Cases2 cases

1 to 2 of 2

1. IL-0541 Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to them

   Twingate2023

   Symptoms

   At 01:40 UTC on 14 February 2023, Twingate saw the number of connections drop in two of its Relay clusters, the one in us-east4 (Ashburn, Virginia) and the one in europe-west6 (Zurich, Switzerland).

   Network unreachable

   Trigger

   At 01:00 UTC on 14 February, Twingate started a Terraform change to all of its Relay clusters worldwide to add spot instance scaling; the rollout took about 45 minutes.

   Root cause

   Configuration change

   Component

   TerraformVPC route tables and firewall rules

   Contributing factor

   Unrepresentative stagingNo staged rollout

   Detection

   At 01:40 UTC Twingate noticed a decrease in the number of connections in the two clusters and started investigating.

   Remediation

   • Disabled the pods in the two affected Relay clusters at 02:04 UTC so Clients and Connectors reconnected to the next closest Relay cluster
   • Corrected the Terraform configuration error and redeployed the affected clusters at 03:30 UTC

   Action items

   • Make the staging environment reflect the sizing and configuration differences between Relay clusters
   • Create Terraform plans automatically for every environment and configuration variation on feature development branches
   • Extend Relay health checks so a cluster is not discoverable to Connectors and Clients when its network firewall tag is missing
   • Research and introduce staggered Terraform rollouts for the Relay cluster infrastructure
   • Add multi-cluster connectivity to Connectors so they handle regional Relay cluster problems automatically

   View case
2. IL-0519 Apps on the cloud.gov PaaS became unavailable after an AWS security group change blocked their outbound traffic

   cloud.gov2016

   Symptoms

   From 9:28am Pacific Time on 8 March 2016, applications on the cloud.gov PaaS that relied on outbound traffic to reach the services they depended on became unavailable, and at one point every application on cloud.gov was down.

   Network unreachableService outage

   Trigger

   At 9:28am Pacific Time on 8 March 2016 a contractor at Aquilent changed the AWS security group settings, one of several changes made in the process of achieving ATO for the platform.

   Root cause

   Configuration change

   Component

   VPC route tables and firewall rules

   Contributing factor

   –

   Remediation

   • Added exceptions for individual applications, which restored some of them
   • Rolled back the security group change, which restored all applications

   Action items

   • Put all security group configuration into version control and make changes through pull requests
   • Monitor the applications running on cloud.gov and alert the cloud.gov team when they do not work as expected
   • Use CF security groups instead of AWS security groups and make the current rules visible to the cloud.gov team and the teams using it
   • Create a notification system to reach the key contact of each application on the platform
   • Create a more accurate staging environment to test changes like this

   View case

By root cause1