Incident Library

IL-0312 Gadget · 2024-12-13database

Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went down

Gadget runs one main Postgres database for all application data, with PgBouncer pooling connections from the API servers' Node.js processes. Gadget had planned for a doubling of traffic over Black Friday / Cyber Monday but saw almost 5x the load, and max_connections was no longer high enough. PgBouncer fetches each user's password with an auth_query at connection time, the only PgBouncer authentication mode Gadget's database vendor supports, so new connections doubled during traffic spikes and caused small disruptions. Raising max_connections requires a full restart. The vendor offers an in-place restart or a failover to a new replica built with the new configuration; because the database holds many terabytes and a replica takes a long time to build, Gadget chose the in-place restart, expecting less downtime.

Postgres shut down at 11:25 AM and failed to come back at 11:26, because months of uptime had fragmented the kernel's memory and the huge_pages allocations could not be met. The cause was identified at 11:45, and a hardware reset of the database server cleared the memory so Postgres started at 11:50. The platform was fully unavailable for 35 minutes. Compute that the autoscalers had removed during the outage then had to be scaled back up almost 4x, and all metrics recovered at 12:10 PM. Shopify webhooks were delayed but not lost, because Shopify retried delivery and Gadget's nightly reconciliation re-ran model actions for missed deliveries.

Severity
–
Scope
All requests to Gadget applications: fully unavailable for 35 minutes after the restart, then elevated latency until 12:10 PM EST; Shopify webhook data delayed but not lost
Users affected
All Gadget applications
Region
–
Time to detect
1m
Time to resolve
45m

Symptoms

At 11:25 AM EST on 13 December 2024, Gadget restarted its main PostgreSQL server in place to raise max_connections. Postgres shut down but did not start again, and every request to Gadget applications failed, including frontend requests, GraphQL API requests, webhook processing and syncs.

The new Postgres process could not start because the kernel could not give it the memory it asked for. Postgres runs with huge_pages=on and allocates many gigabytes for its shared_buffer pools at startup. After months of continuous operation the server's memory had become fragmented, so there was enough free memory in total but not enough contiguous free memory for the huge page allocations. Neither Gadget nor its database vendor knew this could happen, and the cause was found at 11:45 AM.

After the database came back at 11:50 AM, requests were processed with elevated latency until 12:10 PM. During the outage Gadget's compute autoscalers had seen less load and scaled down, so almost 4x the compute had to be provisioned after recovery, which strained the provisioning systems and added about 20 minutes in which some applications' requests were much slower.

The restart was itself a response to earlier, very small disruptions. Behind PgBouncer, each new gadget_api connection briefly opens two Postgres connections, one as superuser for the auth_query that fetches the user's password and one for the pooled connection, so traffic spikes quickly reached max_connections and returned errors to PgBouncer. PgBouncer retries such errors very slowly, so frontend API processes timed out on queries, and an API server whose SELECT 1 health check cannot run fast enough is marked unhealthy and removed from the load balancer.

Symptom Service outageHigh latency

Root cause

Root cause Capacity exhaustion

Component DatabaseLinux kernelDatabase proxy and connection pooler
Contributing factor Unverified assumptionUnrepresentative stagingAutomation amplificationExcessive blast radius
Trigger
An emergency in-place restart of the main Postgres server at 11:25 AM EST, decided at 11:00 AM, to raise max_connections, which Postgres cannot change without a full restart.

Detection

The team carrying out the planned restart saw at 11:26 AM EST that Postgres had shut down and did not start again.

Remediation

Action items

Lessons

Gadget names three deeper factors. Choosing the in-place restart put the database into a state the team had not seen before and was not confident in. Capacity planning for Black Friday / Cyber Monday assumed a doubling of traffic instead of the almost 5x that came, so settings such as max_connections stopped fitting and had to be changed in an emergency instead of in a maintenance window practised in staging. And there was no way to test database configuration changes in a production-like environment: integration tests run against a synthetic Postgres database that never goes through the months of production load behind the memory fragmentation or the memory usage issues.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Linux kernel, Database proxy and connection pooler among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.