IL-0588 AppOptics · 2018-05-31databaseawsobservability

AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper

AppOptics and Librato run their metrics pipeline, including alerting, as stream processing on Apache Kafka, and every incoming data point is written to one or more Kafka topics. The infrastructure runs across several availability zones, including the one hit by the power event, and there is more than one Kafka cluster so that producers can fail over to another cluster when the partition they want to write to is offline. On May 31, 2018 a power event in one availability zone took nodes of several critical systems out of reach, including the ZooKeeper leader that coordinated the Kafka brokers. The ensemble briefly lost quorum and had to elect a new leader, and Kafka brokers in that zone were restarted.

Kafka should have moved partition leadership to the replicas, but many partitions stayed offline without a leader, so the metrics APIs returned 5xx for most incoming data. The multi-cluster setup did not help, because several partitions of the different clusters had ended up in the same availability zones, and the Kafka clusters shared one ZooKeeper ensemble, so coordination broke on both clusters while ZooKeeper had no quorum. AppOptics could not tell how much the loss of quorum affected Kafka's recovery, and it believed Kafka should have regained partition leadership without manual intervention once ZooKeeper had quorum again.

The team restarted one ZooKeeper node to force a leader election (quorum was back by 22:01 UTC), found the leaderless partitions at 22:07 UTC, and restarted every broker in turn until all partitions were online at 23:45 UTC. The postmortem gives the outage as 21:53 to 23:22 UTC. The status page marked the incident resolved at 01:00 UTC on June 1, once the backlog had been processed.

Severity: –
Scope: Metrics APIs of AppOptics and Librato returned 5xx for a majority of incoming metric data; APM metrics and traces from AppOptics APM were affected and many alert notifications were delayed
Users affected: AppOptics and Librato customers
Region: AWS us-east-1
Time to detect: 13m
Time to resolve: 1h 29m

Symptoms

On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

The status page reported errors for metric submission and trace processing delays at 22:06 UTC. In the metrics pipeline, which also drives alerting and writes all incoming data to Kafka topics, many Kafka partitions were offline and had no leader. Writes and reads to those topics stopped, and the API dropped requests with 500 codes.

The ZooKeeper ensemble that coordinates the Kafka brokers had lost its leader node in the power event. The team restarted one ZooKeeper node by hand to force a leader election, and by 22:01 UTC the ensemble had quorum again with a new leader. Kafka brokers in the affected availability zone were restarted as well, but many partitions did not fail over to their replicas and did not recover on their own.

A rolling restart of every Kafka broker brought partitions back a few at a time, and availability improved slowly as each broker came back. Only at 23:45 UTC were all partitions online again. The backlog of alerts was then processed, and the status page warned that this could delay alert notifications. The backlog was fully processed by 00:38 UTC on June 1.

Symptom High error rate Failover failure Monitoring outage

Root cause

Root cause Cloud provider outage

Component Message broker Coordination and key-value store

Contributing factor Single point of failure Excessive blast radius Monitoring blind spot

Trigger: A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

Detection

–

Remediation

Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
Restarted every Kafka broker in a rolling restart until all partitions were back online
Worked through the backlog, including the backlog of alerts, and resumed normal operations

Action items

Isolate the Kafka clusters better across availability zones and regions
Stop sharing one ZooKeeper ensemble across Kafka clusters
Run game days on this failure and check whether newer Kafka versions fix it
Put in place more checks that notify the team earlier about subsystems that need extra help to recover

Lessons

Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

Sources

https://status.appoptics.com/incidents/dbnq4l2063hh · official-status · archived copy

Related cases

IL-0410 SQS Fair Queues rejected valid messages with InvalidParameterValue, failing about half of AssemblyAI's US async requestsAssemblyAI · 2026
same root cause · shared Message broker
IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were downPython Software Foundation · 2022
same root cause · shared Coordination and key-value store
IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degradedPagerDuty · 2013 · sev1
same root cause · shared Coordination and key-value store

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker, Coordination and key-value store among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0588
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0588/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.