Cases15 cases

11 to 15 of 15

1. IL-0280 Kafka brokers in many New Relic cells ran out of disk, leaving US data collection and alerting unusable for hours

   New Relic2021

   Symptoms

   On 29 July 2021, data collection and alerting in New Relic's US region became unusable for many customers for several hours. It began with one unresponsive Kafka broker in a single cell of NRDB, New Relic's telemetry data platform, and over the following hours Kafka brokers in more and more cells failed as their disks filled up.

   Node disk fullThroughput degradationMonitoring outage

   Trigger

   Responders used an incident response tool to extend Kafka data retention for the degraded cell so that its backlog would not be lost. The retention increase was larger than usual, and the tool applied it to all cells of the same type, not only the affected one.

   Root cause

   Operator error

   Component

   Message broker

   Contributing factor

   Harmful manual actionNo staged rolloutSignal buried in noiseUnverified assumption

   Detection

   Engineers saw a Kafka broker in one cell become unresponsive on the morning of 29 July. Later a monitoring alert warned that disk space was running low, but it fired together with unrelated warnings from several other systems and was missed.

   Remediation

   • Restarted the unresponsive Kafka broker by hand, while the automated restart was also running
   • Resolved the disk space problem on the Kafka brokers, after which service recovered slowly

   Lessons

   Cell isolation is the strongest protection against a widespread disruption, and emergency tools have to respect it just as normal operations do. Emergency response tools must be safe to use in chaotic situations without losing their ability to work. Engineers' attention is limited, so incident response processes and alerts need regular review to direct it at the most important signal.

   View case
2. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers

   CircleCI2021

   Symptoms

   From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.

   Throughput degradationCloud API rate limit exhaustedDatabase overload

   Trigger

   The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.

   Root cause

   Breaking change in an upgrade

   Component

   Message brokerDatabase

   Contributing factor

   Change with a delayed effectIgnored early signalRetry stormExcessive blast radiusOverlapping failures

   Detection

   Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.

   Remediation

   • Blocked Linux machine jobs from entering the work queue
   • Had GCP raise the CPU quotas
   • Deleted old VMs by hand
   • Cancelled some workflows to free resources
   • Reset the VM services deployment so the VM destroy queues had consumers again
   • Scaled up the RDS database instance, at 18:20 and again at 19:15
   • Turned down VM provisioning and blocked remote-docker jobs to shed load
   • Scaled down all pods that created and destroyed VMs
   • Deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues
   • Brought provisioning and distribution back one component at a time and reversed the mitigation code changes

   Action items

   • Make VM provisioning more robust to changes in load so it can recover from thundering herds
   • Make VM provisioning independent so its failure does not affect Docker jobs
   • Review and improve observability of RabbitMQ and the production queues
   • Batch requests to avoid hitting GCP API rate limits
   • Improve incident response for long-running incidents

   View case
3. IL-0384 Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failed

   Amazon Web Services2020

   Symptoms

   At 5:15 AM PST on 25 November 2020, alarms fired for errors on putting and getting Amazon Kinesis records in the Northern Virginia (US-EAST-1) Region. From the same time CloudWatch, which processes metric and log data through Kinesis Data Streams, and CloudWatch Events and EventBridge returned more errors.

   High error rateHigh latencyMonitoring outage

   Trigger

   A relatively small addition of capacity to the Kinesis front-end fleet, made between 2:44 AM and 3:47 AM PST.

   Root cause

   Capacity exhaustion

   Component

   Message brokerServerless function platformTelemetry agent

   Contributing factor

   Unknown limitExcessive blast radiusCircular dependency

   Detection

   Alarms for errors on putting and getting Kinesis records fired at 5:15 AM PST, and the teams began reviewing logs.

   Remediation

   • Removed the newly added front-end capacity
   • Added a configuration so front-end servers load bootstrap data from the authoritative metadata store instead of neighbouring servers
   • Restarted the front-end fleet gradually, a few hundred servers per hour from 10:07 AM PST
   • Added Cognito capacity to buffer more calls to Kinesis
   • Deployed a change that reduced Cognito's dependency on Kinesis
   • Mitigated the memory contention on the Lambda invocation hosts
   • Informed customers through the backup Service Health Dashboard tool, the Personal Health Dashboard and a global banner

   Action items

   • Move the Kinesis front-end to larger CPU and memory servers so fewer servers, and fewer threads per server, are needed
   • Add fine-grained alarming for thread consumption
   • Finish testing a higher operating system thread count limit
   • Greatly improve the cold-start time of the front-end fleet
   • Move the front-end server cache to a dedicated fleet
   • Move a few large AWS services such as CloudWatch to a separate, partitioned front-end fleet
   • Accelerate the cellularization of the front-end fleet to match the back-end
   • Change the Cognito webservers so they sustain Kinesis API errors without exhausting their buffers
   • Keep 3 hours of metric data in the CloudWatch local metrics data store, first in US-EAST-1 and then globally
   • Train support engineers regularly on the backup tool for posting to the Service Health Dashboard

   Lessons

   In the Kinesis front-end, the number of threads each server keeps grows directly with the number of servers in the fleet, so adding servers can push every server past an operating system limit at the same time. AWS sees cellularization of the front-end, already done for the back-end, as the way to keep the shard-map cache within a tested and operated range and to guard against unknown scaling limits.

   View case
4. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper

   AppOptics2018

   Symptoms

   On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

   High error rateFailover failureMonitoring outage

   Trigger

   A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

   Root cause

   Cloud provider outage

   Component

   Message brokerCoordination and key-value store

   Contributing factor

   Single point of failureExcessive blast radiusMonitoring blind spot

   Remediation

   • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
   • Restarted every Kafka broker in a rolling restart until all partitions were back online
   • Worked through the backlog, including the backlog of alerts, and resumed normal operations

   Action items

   • Isolate the Kafka clusters better across availability zones and regions
   • Stop sharing one ZooKeeper ensemble across Kafka clusters
   • Run game days on this failure and check whether newer Kafka versions fix it
   • Put in place more checks that notify the team earlier about subsystems that need extra help to recover

   Lessons

   Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

   View case
5. IL-0275 Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hours

   Parse.ly2015

   Symptoms

   On Thursday night, 26 March 2015, and again on Friday, 27 March, Parse.ly's data processing backend stopped. Monitoring tools such as Munin, Sentry and Pingdom reported failures widely, PagerDuty woke the on-call engineer, and no new analytics data appeared in the dashboards at dash.parsely.com and preview.parsely.com for several hours.

   Service outageDNS resolution failures

   Trigger

   Data volume reached a point where the Kafka brokers, running on modest Amazon EC2 nodes without high network capacity, hit operating system network limits under the outbound read traffic and became unavailable.

   Root cause

   Capacity exhaustion

   Component

   Message brokerService discovery

   Contributing factor

   Monitoring blind spotUnknown limitNo headroomHarmful manual action

   Detection

   Munin, Sentry and Pingdom reported the failures widely, Graphite graphs showed them, and PagerDuty woke the on-call engineer when the first outages started in the night.

   Remediation

   • Added new brokers to expand the Kafka cluster
   • Corrected the configuration of the new brokers and deployed them to the cluster, after which network utilization per node dropped and processing resumed
   • Turned the downstream consumer systems back on slowly through Apache Storm and reprocessed the backlog
   • Began restoring the data to customer accounts from the raw data archived in Amazon S3

   Lessons

   CPU and disk I/O are good signs of machine health, but creeping network utilization is harder to monitor, because 100% of network capacity is not well defined on cloud VMs that share host machines. When every node of a cluster runs close to the same limit, losing one node can take down the whole cluster, so failure planning should cover an entire subsystem going offline, not just one or two machines. Keeping raw data archival in simple files and S3 uploads, separate from the Kafka stream, meant there was no risk of permanent data loss.

   View case

By root cause8