Incident Library

IL-0265 Open Build Service · 2023-01-05networking

Open Build Service returned 503 errors for about two hours after an HAProxy package update changed its configuration

OBS runs updates during its maintenance windows on Thursdays from 8AM to 10AM CET/CEST. On 5 January 2023 an update started at 10:26 UTC, after the window, because its end was wrongly assumed to be 11AM. The updated haproxy package included configuration changes the team had not expected, and OBS went down at 10:27. The secondary HAProxy system could not take over, since the automated sync of renewed certificates to it had stopped without notice; OBS deploys with Salt, but the certificates of the secondary machines had been missed.

The team adapted all the HAProxy configuration files from 10:47, but the service stayed down because a "MAINTENANCE" flag left in the Apache configuration on another server kept that Apache serving the maintenance page instead of starting Passenger. At 12:18 the team removed the flag and restarted the server, and OBS was up again at 12:20. No permanent damage was done.

Severity
–
Scope
OBS reference server: API and web interface unavailable, answering with a maintenance message or HTTP 503
Users affected
everyone using the OBS reference server
Region
–
Time to detect
0m
Time to resolve
1h 53m

Symptoms

On 5 January 2023 the Open Build Service (OBS) reference server went offline for about two hours, one minute after a package update started at 10:26 UTC. The application answered with a maintenance message or an HTTP 503 error (Service Unavailable), and nobody could work with the API or the web interface.

The updated haproxy package had brought configuration changes that the team did not expect. Switching to the secondary HAProxy system at 10:35 did not help, because that system's certificates were out of date: the automated sync of renewed certificates had stopped without any notice. The team started adapting the configuration at 10:47.

After the configuration files were adapted, the service still did not come back, and from 11:27 the team was debugging why. A "MAINTENANCE" flag had been left in the Apache configuration on another server, a machine that is usually not run with that flag. With the flag set, that Apache showed the maintenance page and did not start Passenger, even though the Apache daemon itself started without visible problems.

Symptom Service outage

Root cause

Root cause Breaking change in an upgrade

Component HAProxy
Contributing factor Overlapping failuresMonitoring blind spotConfiguration not in code
Trigger
The haproxy package on the reference server was updated with zypper; the update started after the Thursday maintenance window had ended, because the window was wrongly assumed to end at 11AM.

Detection

Both the team and customers saw that the application was down right after the update; customers reported it through different communication channels.

Remediation

Action items

Lessons

The OBS team wants to analyse the consequences of running "zypper up" more carefully, because some upgrades need attention. Secondary machines and their configuration need watching too: although everything is usually deployed with Salt, the certificates for the secondary machines were missed. When Passenger is started through Apache, it will not start if Apache has problems, so the whole Apache configuration should be checked even when the daemon starts without visible errors. The team also noted to expect the unexpected after a long vacation break.

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, HAProxy among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.