Incident Library
Applied filters Root causeConfiguration changeComponentTerraformSymptomNetwork unreachableClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0541 Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to them
    Twingate2023
    Symptoms

    At 01:40 UTC on 14 February 2023, Twingate saw the number of connections drop in two of its Relay clusters, the one in us-east4 (Ashburn, Virginia) and the one in europe-west6 (Zurich, Switzerland).

    Network unreachable
    Trigger

    At 01:00 UTC on 14 February, Twingate started a Terraform change to all of its Relay clusters worldwide to add spot instance scaling; the rollout took about 45 minutes.

    Root cause
    Configuration change
    Component
    TerraformVPC route tables and firewall rules
    Contributing factor
    Unrepresentative stagingNo staged rollout
    Detection

    At 01:40 UTC Twingate noticed a decrease in the number of connections in the two clusters and started investigating.

    Remediation
    • Disabled the pods in the two affected Relay clusters at 02:04 UTC so Clients and Connectors reconnected to the next closest Relay cluster
    • Corrected the Terraform configuration error and redeployed the affected clusters at 03:30 UTC
    Action items
    • Make the staging environment reflect the sizing and configuration differences between Relay clusters
    • Create Terraform plans automatically for every environment and configuration variation on feature development branches
    • Extend Relay health checks so a cluster is not discoverable to Connectors and Clients when its network firewall tag is missing
    • Research and introduce staggered Terraform rollouts for the Relay cluster infrastructure
    • Add multi-cluster connectivity to Connectors so they handle regional Relay cluster problems automatically
    View case
Check your cluster for this

This case involves Terraform and VPC route tables and firewall rules. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
TerraformVPC route tables and firewall rules
Contributing factor
No staged rolloutUnrepresentative staging
Organisation
Twingate
  1. IL-0541 Two Twingate Relay clusters were discoverable but unreachable, so Clients and Connectors could not attach to them
    Twingate2023