Cases2 cases

1 to 2 of 2

1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline

   UK Ministry of Justice2024

   Symptoms

   On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.

   Service outageNetwork unreachableResources deleted

   Trigger

   At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.

   Root cause

   Operator error

   Component

   TerraformCI systemVPC route tables and firewall rules

   Contributing factor

   No safeguard on destructive change

   Detection

   High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.

   Remediation

   • Ran terraform apply locally to rebuild the subnet route table associations

   Action items

   • Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
   • Find a way to hold queued pull requests while manual infrastructure updates are under way
   • Find ways to isolate the plan and apply tasks of infrastructure pipelines better

   View case
2. IL-0541 Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to them

   Twingate2023

   Symptoms

   At 01:40 UTC on 14 February 2023, Twingate saw the number of connections drop in two of its Relay clusters, the one in us-east4 (Ashburn, Virginia) and the one in europe-west6 (Zurich, Switzerland).

   Network unreachable

   Trigger

   At 01:00 UTC on 14 February, Twingate started a Terraform change to all of its Relay clusters worldwide to add spot instance scaling; the rollout took about 45 minutes.

   Root cause

   Configuration change

   Component

   TerraformVPC route tables and firewall rules

   Contributing factor

   Unrepresentative stagingNo staged rollout

   Detection

   At 01:40 UTC Twingate noticed a decrease in the number of connections in the two clusters and started investigating.

   Remediation

   • Disabled the pods in the two affected Relay clusters at 02:04 UTC so Clients and Connectors reconnected to the next closest Relay cluster
   • Corrected the Terraform configuration error and redeployed the affected clusters at 03:30 UTC

   Action items

   • Make the staging environment reflect the sizing and configuration differences between Relay clusters
   • Create Terraform plans automatically for every environment and configuration variation on feature development branches
   • Extend Relay health checks so a cluster is not discoverable to Connectors and Clients when its network firewall tag is missing
   • Research and introduce staggered Terraform rollouts for the Relay cluster infrastructure
   • Add multi-cluster connectivity to Connectors so they handle regional Relay cluster problems automatically

   View case

By root cause2