Cases1 case

1 to 1 of 1

1. IL-0546 Buildkite jobs could not go to newly connected agents once agent IDs passed a 32-bit integer foreign key's limit

   Buildkite2023

   Symptoms

   From 10:06:20 UTC on Saturday 22 July 2023, Buildkite agents that connected could not pick up jobs, while agents that had connected before that moment kept working. Automated monitoring alerted the team within 4 minutes.

   Integer ID overflow

   Trigger

   At 10:06:20 UTC on 22 July 2023, the primary key for connected agents went past the maximum value of the PostgreSQL integer type.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Monitoring blind spot

   Detection

   Automated monitoring alerted the team within 4 minutes of the ID passing the integer maximum.

   Remediation

   • Changed the agent ID sequence at 11:40 UTC to start at -1 and increment by -1, after verifying it in non-production environments and reconfiguring table partitioning
   • Disconnected the agents that had connected between 10:06 and 11:40 UTC by 12:08 UTC so they reconnected with new IDs

   Action items

   • Monitor identifiers and alert when one exceeds 50% of its maximum value
   • Audit all existing primary and foreign keys for correctness and remaining runway
   • Add ratchet tests that stop new 32-bit integer identifiers from being introduced

   Lessons

   Buildkite said that, while existing numeric primary keys stay in use, the incident showed a blind spot in its monitoring.

   View case

By root cause1