Incident Library

IL-0546 Buildkite · 2023-07-22database

Buildkite jobs could not go to newly connected agents once agent IDs passed a 32-bit integer foreign key's limit

Buildkite stores connected agents in PostgreSQL with a `bigint` primary key, but a foreign key that maps build jobs to agents was stored as `integer`. At 10:06:20 UTC on 22 July 2023 the agent ID passed 2,147,483,647, the largest value a signed 32-bit integer can hold, and from then on jobs could not be assigned to any agent that connected. Monitoring alerted within 4 minutes. The team started migrating the foreign key to `bigint` and looked for a faster fix at the same time. Neither the SQL standards nor PostgreSQL offer unsigned integer types, but PostgreSQL sequences accept a negative start and a negative increment and Rails/ActiveRecord handles that, so the negative half of the 32-bit range could serve as a temporary workaround. The IDs are internal only, since Buildkite uses separate UUIDs for public references, so no outside system would reject negative IDs.

After testing the workaround in non-production environments and reconfiguring the table partitioning, the team changed the agent ID sequence at 11:40 UTC to start at -1 and step by -1, and agents that connected from then on worked. By 12:08 UTC the team had disconnected the agents that had connected between 10:06 and 11:40, so they came back with new IDs. Buildkite already used time-ordered UUID primary keys for new tables as part of its database sharding project, but is keeping existing numeric primary keys for some time, and the incident showed a blind spot in its monitoring.

Severity
–
Scope
Agents connected between 10:06:20 and 11:40:09 UTC could not pick up new jobs; the status page lists Agent API and Job Queue
Users affected
–
Region
–
Time to detect
4m
Time to resolve
2h 2m

Symptoms

From 10:06:20 UTC on Saturday 22 July 2023, Buildkite agents that connected could not pick up jobs, while agents that had connected before that moment kept working. Automated monitoring alerted the team within 4 minutes.

The cause was in PostgreSQL: the internal primary key for connected agents had just passed 2,147,483,647, the maximum of the PostgreSQL `integer` type (signed 32-bit). The primary key itself was a `bigint` and had room to spare, but some tables referenced agent IDs through foreign key columns of type `integer`, and the IDs of agents created after 10:06:20 could not be stored in those columns. The status page reported the problem first as newly connected agents being unable to run jobs, and later as tables trying to reference 64-bit agent IDs with 32-bit foreign key columns.

Symptom Integer ID overflow

Root cause

Root cause Capacity exhaustion

Component Database
Contributing factor Monitoring blind spot
Trigger
At 10:06:20 UTC on 22 July 2023, the primary key for connected agents went past the maximum value of the PostgreSQL integer type.

Detection

Automated monitoring alerted the team within 4 minutes of the ID passing the integer maximum.

Remediation

Action items

Lessons

Buildkite said that, while existing numeric primary keys stay in use, the incident showed a blind spot in its monitoring.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.