Cases2 cases

1 to 2 of 2

1. IL-0173 cert-manager's preferredChain logic picked the Let's Encrypt chain through the expired DST Root CA X3, so clients built on an old BoringSSL could not connect to a development server

   Devsisters2024

   Symptoms

   On 13 February 2024, when Devsisters' infrastructure cell came back from the Lunar New Year holiday, it received a request to look into a development server that could not be reached. No one had made a manual change over the holiday.

   TLS certificate errors

   Trigger

   On 8 February 2024, in the first phase of retiring the DST Root CA X3 cross-sign, Let's Encrypt made the ISRG Root X1 chain its default chain and began offering the DST Root CA X3 chain as the alternate.

   Root cause

   Software defect

   Component

   cert-manager

   Contributing factor

   –

   Detection

   On 13 February, after the Lunar New Year holiday, a request came in to the infrastructure cell that the development server could not be reached; while inspecting the certificate chain the server presented, the team found it serving the chain through the expired DST Root CA X3.

   Remediation

   • submitted a pull request fixing cert-manager's preferred chain selection, which was merged

   Lessons

   A latent bug can stay hidden until an upstream provider changes a default. cert-manager's preferredChain matching gave the intended result while DST Root CA X3 was Let's Encrypt's default chain and the wrong one after the default changed. An expired root at the top of a served chain is harmless to most TLS clients, which fall back to the trusted ISRG Root X1 below it, but fatal to an implementation such as an old BoringSSL that checks the trust anchor's validity and insists on verifying the topmost certificate. ACME clients choose the preferred chain with different heuristics, so the same preferredChain value can behave differently between Certbot and cert-manager.

   View case
2. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured

   Devsisters2021

   Symptoms

   During a team dinner, a Slack alert reported that one node of the CockroachDB cluster behind Cookie Run: Kingdom had gone down. The DevOps engineers were not worried at first, because CockroachDB is built to survive the loss of some nodes.

   Node NotReadyData lossService outage

   Trigger

   A cooling system failure in an AWS Tokyo region data centre overheated the servers in one availability zone, which held CockroachDB nodes.

   Root cause

   Cloud provider outage

   Component

   Application databaseHelm

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A Slack alert reported that a database node had gone down.

   Remediation

   • copied the data outside the broken ranges into new tables with INSERT INTO ... SELECT queries that skipped the affected ranges
   • ran debug unsafe-remove-dead-replicas together with Cockroach Labs engineers, recovering one of the two lost data ranges
   • restored the last data range with the storage-layer recovery method from the previous incident

   Action items

   • Configure CockroachDB Locality so data is spread evenly across availability zones
   • Deploy with one chart and a separate values file per availability zone to set Locality
   • Replace nodes by adding Locality-configured nodes, confirming replicas in every zone, then retiring the old nodes

   Lessons

   A replication factor sized for random instance retirements does not protect against a whole availability zone failing; for a game of this scale a multi-AZ placement is required, not optional. Keeping one uniform Helm chart made zone topology hard to set, which is part of why it was left out. A repetitive chart with a values file per zone sets it reliably.

   View case

By root cause2