Incident Library
Applied filters Root causeBreaking change in an upgradeFactorMonitoring blind spotOrganisationOpen Build ServiceClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0265 Open Build Service returned 503 errors for about two hours after an HAProxy package update changed its configuration
    Open Build Service2023
    Symptoms

    On 5 January 2023 the Open Build Service (OBS) reference server went offline for about two hours, one minute after a package update started at 10:26 UTC. The application answered with a maintenance message or an HTTP 503 error (Service Unavailable), and nobody could work with the API or the web interface.

    Service outage
    Trigger

    The haproxy package on the reference server was updated with zypper; the update started after the Thursday maintenance window had ended, because the window was wrongly assumed to end at 11AM.

    Root cause
    Breaking change in an upgrade
    Component
    HAProxy
    Contributing factor
    Overlapping failuresMonitoring blind spotConfiguration not in code
    Detection

    Both the team and customers saw that the application was down right after the update; customers reported it through different communication channels.

    Remediation
    • Adjusted all the HAProxy configuration files
    • Removed the maintenance mode flag from the Apache configuration
    • Restarted the server
    Action items
    • Discuss whether a maintenance configuration block is a benefit or a problem
    • Review the synchronization of SSL certificates for the secondary proxies
    Lessons

    The OBS team wants to analyse the consequences of running "zypper up" more carefully, because some upgrades need attention. Secondary machines and their configuration need watching too: although everything is usually deployed with Salt, the certificates for the secondary machines were missed. When Passenger is started through Apache, it will not start if Apache has problems, so the whole Apache configuration should be checked even when the daemon starts without visible errors. The team also noted to expect the unexpected after a long vacation break.

    View case
Check your cluster for this

This case involves HAProxy. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
HAProxy
Contributing factor
Configuration not in codeMonitoring blind spotOverlapping failures
Organisation
Open Build Service
  1. IL-0265 Open Build Service returned 503 errors for about two hours after an HAProxy package update changed its configuration
    Open Build Service2023