Cases5 cases

1 to 5 of 5

1. IL-0568 A layer 2 loop after an access switch change broke network traffic across a whole DeNA site, with mass MACFLAP logs

   DeNA2026

   Symptoms

   At about 20:00 on a weekday, one of DeNA's largest sites reported that terminals of a customer-facing system could not communicate and business was suffering. The network group's monitoring had raised no alert.

   Intermittent connection failuresHigh latency

   Trigger

   During a routine change on 15 Cisco Catalyst 2960L access switches at the site, which tidied descriptions, added VLAN definitions and added allowed VLANs on trunk ports, `no switchport trunk allowed vlan` was applied to GigabitEthernet0/15 of switch SW3, only one of the two physical ports in the LACP bundle between SW3 and SW2.

   Root cause

   Operator error

   Component

   Network switch

   Contributing factor

   Monitoring blind spotNo staged rolloutNo runbookChange during peak hours

   Detection

   The site reported the problem at about 20:00; no monitoring alert fired, because the affected port had no interface monitoring and LACP monitoring had been removed by an earlier monitoring change.

   Remediation

   • Rolled back the switch changes without knowing the cause; service recovered when the interface setting of one switch was restored
   • Found the faulty port and confirmed there was no other impact, completing the first response at about 22:00

   Action items

   • Fix the missing interface and LACP monitoring
   • Require a review for every monitoring change
   • Block merges to the alert rule repository without a reviewer's approval
   • Enable automatic AI review on pull requests
   • Announce every network change in advance, however minor
   • Agree the work date with the site's managers and avoid the site's event days
   • Prepare rollback procedures for high-impact maintenance and for work on several devices at once

   Lessons

   Any change can cause an outage, through a mistake, an unexpected event or a bug, so changes need to be scheduled with a maintenance window agreed in advance. A change on a downstream access switch can affect the whole site. Changing many devices at once made it slow both to find the faulty device and to roll back, and the parts that can be prevented by mechanisms rather than by people's care, such as reviews and monitoring, should be enforced that way.

   View case
2. IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturated

   Scaleway2025

   Symptoms

   On 10 March 2025 at 16:22 UTC, public internet connectivity in Scaleway's FR-PAR region broke for all products. Most internet traffic between the FR-PAR-2 availability zone and the rest of the PAR region was blackholed for about five minutes, while VPC traffic and Dedibox RPN kept working.

   Packet lossService outageHigh latency

   Trigger

   While Scaleway's network team connected a new data center to the FR-PAR backbone, a human mistake in a router configuration rerouted more than 1 Tbps of traffic between OpCore DC3 and DC5 through a single 100 Gbps link.

   Root cause

   Operator error

   Component

   Backbone network (WAN)Object storageAuthoritative DNS servers and records

   Contributing factor

   Automation amplification

   Detection

   The router configuration error was detected quickly and rolled back within four minutes.

   Remediation

   • Rolled back the router configuration, restoring the correct configuration at 16:27 UTC

   Action items

   • Harden the peer review and production communication process for changes
   • Add failsafes and tests before network configuration is applied at backbone level
   • Deploy the fix for the Object Storage synchronisation software issue

   View case
3. IL-0582 Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growing

   Kanmu2024

   Symptoms

   A Datadog alert fired because a large number of unprocessed jobs had piled up in Kanmu's job queue and were not being worked off. The jobs were the batch enqueued at about the same time every day, in the usual volume, but the workers took far longer than normal to dequeue and process them.

   Throughput degradationHigh latency

   Trigger

   A heavy ad hoc analysis query from the BI tool Redash ran for a long time on the writer instance, because the Redash data source pointed at the cluster endpoint.

   Root cause

   Operator error

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Change with a delayed effectUnreviewed defaults

   Detection

   A Datadog alert on unprocessed jobs piling up in the job queue.

   Remediation

   • Changed the Redash data source to the reader endpoint
   • Ended the long-running Redash query with pg_terminate_backend
   • Ran VACUUM on que_jobs by hand again, which cleaned up the dead tuples

   Action items

   • Make Redash unable to connect to the writer instance even if the writer endpoint is set, by mapping its host name to a TEST-NET-1 address in the Redash container's /etc/hosts
   • Set statement_timeout on PostgreSQL users so that long queries stop automatically
   • Consider improving qg's dequeue algorithm or adopting a job queue such as river that uses SELECT FOR UPDATE SKIP LOCKED

   Lessons

   A long-running query on the same database instance, even one unrelated to que_jobs, can keep VACUUM from removing que_jobs dead tuples, so the table keeps growing and the job queue slows down. qg's design with advisory locks and a recursive CTE is sensitive to dead tuples.

   View case
4. IL-0434 Rolling restarts of a 30-node Elasticsearch 0.19 cluster after a host failure left 12 shards unassigned

   CyberAgent2016

   Symptoms

   After an OpenStack host server went down and took one Elasticsearch node with it, the health check that the API servers run against Elasticsearch returned false, although the Head plugin showed the cluster status as green. The check stayed false after the node was brought back and the cluster was green again.

   High latencyData loss

   Trigger

   An OpenStack host server went down, and one node of the Elasticsearch cluster went down with it.

   Root cause

   Operator error

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   Unverified assumptionAutomation amplification

   Detection

   The health check that the API servers run against Elasticsearch returned false while the Head plugin showed the cluster as green.

   Remediation

   • Migrated the Elasticsearch node's virtual server to another host after the failed host recovered
   • Started the node's Elasticsearch process and let it rejoin the cluster
   • Restarted all nodes in order from node 1, pausing at about the twentieth when the API slowed
   • Restarted all nodes again from node 1, which ended the API delays but left 12 shards unassigned
   • Tried to move an unassigned shard with the allocate command of _cluster/reroute, which failed
   • For index A, copied node 22's shard files to nodes 3 and 10 and set primary to true in node 22's state file
   • For index B, deleted the broken shard files on node 27, copied node 28's files to node 27 and other nodes and marked node 28 primary
   • For index C, copied shards 2 and 4 from nodes 27 and 25 to two other nodes each and marked the copies as not primary

   Action items

   • Upgrade Elasticsearch and stop running such an old version

   Lessons

   Automatic shard reallocation should have been turned off before restarting nodes. After each node restart, the next node should have been restarted only after reallocation had completely finished, not just after the log file looked fine.

   View case
5. IL-0224 ELB control plane APIs failed in US-East, and load balancers it then changed broke, after ELB state data was deleted

   Amazon Web Services2012

   Symptoms

   From 12:24 PM PST on 24 December 2012, the Amazon Elastic Load Balancing (ELB) control plane in the US-East Region showed high latency and error rates for API calls that manage load balancers. Running load balancers kept handling requests normally at first.

   Control plane lagHigh error rateHigh latency

   Trigger

   One of the very few developers with access to the production ELB environment inadvertently ran a maintenance process against the production ELB state data, and it logically deleted part of that data.

   Root cause

   Operator error

   Component

   Cloud load balancer

   Contributing factor

   No safeguard on destructive changeUntested backupAutomation amplification

   Detection

   The ELB team noticed the control plane API errors first; the missing state data was found only after the team investigated load balancers that had become degraded.

   Remediation

   • Disabled several ELB control plane workflows, including scaling and descaling
   • Recovered some of the affected running load balancers by hand
   • Restored a snapshot of the ELB state data from just before the deletion, using an alternate recovery process
   • Merged the restored data with the state changes since the snapshot and verified it
   • Re-enabled the ELB workflows and APIs slowly while recovering each affected load balancer

   Action items

   • Require per-incident Change Management approval for all access to production ELB state data
   • Change the data recovery process so ELB state data can be recovered much faster
   • Reprogram the ELB control plane workflows to reconcile central service data with current load balancer state

   View case

By root cause1