IL-0393 BBC · 2014-07-19database

Load on the BBC programme metadata database surged and metadata requests failed, leaving BBC iPlayer down over a weekend

The BBC's programme and clip metadata system has 58 application servers and 10 database servers, split across two data centres in a hot-hot configuration so it can run from either one. It feeds BBC iPlayer applications on more than 1200 supported devices and programme and clip modules across BBC Online. On the morning of 19 July 2014 the database load rose sharply and many metadata requests failed; products without their own cache of the metadata failed with it. At almost the same time a pool of the caching layer in front of BBC Online failed, repeatedly, making BBC iPlayer and the homepage inaccessible, and the BBC restored the caching layer first and put up the emergency homepage several times over the weekend.

System performance stayed poor, so the BBC did significant remedial work on the metadata system from Saturday afternoon into the evening, during which BBC iPlayer was effectively not usable. Afterwards most of the site ran close to normal but BBC iPlayer stayed down on a number of devices, and the BBC kept it that way through the weekend. The full restoration was moved to Monday morning to avoid the Sunday evening peak. Programmes aired on 12 and 13 July were not available in catch-up for some users. At the time of writing the BBC was still investigating the root causes of both failures.

Severity: –
Scope: BBC iPlayer, BBC iPlayer Radio and audio and video playback on bbc.co.uk; the emergency homepage was used for long periods; programmes aired on 12 and 13 July were unavailable in catch-up for some users
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

At 9.30 on Saturday 19 July 2014, the load on the database of the BBC system that provides programme and clip metadata rose sharply, and many requests for metadata to its application servers started to fail. BBC iPlayer, BBC iPlayer Radio and audio and video playback on other parts of bbc.co.uk were affected.

How each product failed depended on how it used the metadata. Products that cache the metadata kept serving content while trying to revalidate, while products that use the metadata directly, mostly older applications, started to fail. At almost the same time, one pool of the caching layer in front of most BBC Online products failed, which made BBC iPlayer and the BBC homepage inaccessible and led to the emergency homepage reading "Due to technical problems, we are displaying a simplified version of the BBC Homepage". Finding the source of the additional database load was far from straightforward, and the metadata service could not be brought back simply by rebooting it.

Symptom Database overload High error rate Service outage

Root cause

Root cause unknown

Component Database

Contributing factor No caching Overlapping failures

Trigger: –

Detection

–

Remediation

Restored the failed pool of the caching layer first
Switched the BBC homepage to its emergency mode until the cache was stable
Did significant remedial work on the metadata system from Saturday afternoon into the evening
Ran the service in a reduced state through the weekend and did the full restoration on Monday morning

Action items

Complete the forensics on the root causes and put measures in place to reduce the chance of similar interruptions

Lessons

–

Sources

https://www.bbc.co.uk/blogs/internet/entries/a37b0470-47d4-3991-82bb-a7d5b8803771 · blog · archived copy

Related cases

IL-0545 Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodesWasabi · 2024
same root cause · shared Database
IL-0441 A ZOZO SQL Server database stalled and applications hit timeout storms whenever batch executions peaked in salesZOZO · 2021
same root cause · shared Database
IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutesStack Exchange · 2017
same root cause · shared Database
IL-0387 Mailgun MongoDB secondaries were overloaded by rising connections, and a rollback then moved reads onto the primary tooMailgun · 2017
same root cause · shared Database

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0393
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0393/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.