Cases39 cases

34 to 38 of 39

1. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of data

   Hatena2017

   Symptoms

   On 7 August 2017, while checking the application after the maintenance that moved Mackerel from an on-premises data center to AWS, the team found that data updated during a specific time window was missing. Everything written to the PostgreSQL database between 14:30 and 15:30 JST was gone.

   Data lossReplication lagNetwork unreachable

   Trigger

   One hour after the application was switched to the newly promoted primary db001, db001 lost network connectivity and the database failed over to the standby db002.

   Root cause

   Operator error

   Component

   DatabaseDatabase replicationDatabase failover manager

   Contributing factor

   Overlapping failuresUnreviewed defaultsUnrepresentative stagingAutomation amplification

   Detection

   The team found the missing data while checking how the application behaved after the migration work.

   Remediation

   • Tried to restore the lost updates by hand from the table data left on db001 and gave up
   • Tried point-in-time recovery from the base backup and WAL and gave up
   • Built a new replica db003 with db002 as primary to restore redundancy
   • Kept db001 in maintenance state for investigation
   • Watched db002 for some time to confirm the problem did not recur, then completed the migration

   Action items

   • Change the PostgreSQL replica creation script to set recovery_target_timeline = 'latest' in recovery.conf by default
   • Add a check to a server configuration test tool such as Serverspec that recovery.conf contains recovery_target_timeline = 'latest'

   Lessons

   A setting needed only for a rare layout, such as a grandchild replica during a migration, was applied by hand through a script option and was forgotten. Because recovery_target_timeline = 'latest' is harmless on ordinary standbys too, Hatena now sets it by default whenever a replica is created. The staging rehearsal used the same layout and steps but had the setting right, so it could not catch the mistake. Hatena notes that a similar layout may be used again for a future PostgreSQL upgrade or a move to another region.

   View case
2. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h

   Google Cloud2017

   Symptoms

   On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.

   Network unreachableProbe failures

   Trigger

   A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.

   Root cause

   Software defect

   Component

   Cloud load balancerNetwork control plane (SDN)

   Contributing factor

   Excessive blast radiusGap in automated tests

   Remediation

   • Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
   • Applied the fix zone by zone between 11:36 and 12:42

   Action items

   • Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
   • Improve the inefficient code path
   • Write new tests that check behaviour on a wider range of configurations
   • Accelerate replacing global propagation of address configuration with decentralized routing
   • Add metrics and alerting that identify this kind of issue sooner

   View case
3. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung

   Netlify2016

   Symptoms

   At 8am PST on Monday, August 22, 2016, Netlify got alerts that its central load balancer, the single public IP that users point A records at, had stopped responding. Rackspace had null-routed the load balancer, so all traffic to that IP hung before it reached Netlify's servers.

   Network unreachable

   Trigger

   A DDoS attack against Netlify's central load balancer in Rackspace's Chicago datacenter, after which Rackspace null-routed the load balancer's public IP.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancer

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts at 8am PST reported that the central load balancer had stopped responding, and Rackspace then confirmed it had null-routed the load balancer.

   Remediation

   • Started a new cluster of CDN machines in Google Compute behind a Google network load balancer
   • Gave clients who had contacted support the new public IP and told them to change their A records
   • Updated the documentation and announced the new IP on Twitter, Gitter and the status page

   Action items

   • Keep the new Google-based network IP running in place of the Rackspace load balancer
   • Release Netlify's own DNS hosting, integrated with the CDN, so domains do not depend on a fixed IP
   • Fix the status page publishing process so it is updated straight away during an issue
   • Build a mailer that sends targeted mails to all paying clients during an outage

   Lessons

   Netlify judged Rackspace's load balancer too vulnerable to DDoS and Rackspace's DDoS processes not up to the standard of other cloud providers, and expected Google's anycast-based Maglev load balancer to be more resilient. In the end, the only way to protect a domain from DDoS-related outages is not to expose any fixed IP address for it, which is why sites should use the www domain with a CNAME, or Netlify DNS, rather than a single A-record IP.

   View case
4. IL-0519 Apps on the cloud.gov PaaS became unavailable after an AWS security group change blocked their outbound traffic

   cloud.gov2016

   Symptoms

   From 9:28am Pacific Time on 8 March 2016, applications on the cloud.gov PaaS that relied on outbound traffic to reach the services they depended on became unavailable, and at one point every application on cloud.gov was down.

   Network unreachableService outage

   Trigger

   At 9:28am Pacific Time on 8 March 2016 a contractor at Aquilent changed the AWS security group settings, one of several changes made in the process of achieving ATO for the platform.

   Root cause

   Configuration change

   Component

   VPC route tables and firewall rules

   Contributing factor

   –

   Remediation

   • Added exceptions for individual applications, which restored some of them
   • Rolled back the security group change, which restored all applications

   Action items

   • Put all security group configuration into version control and make changes through pull requests
   • Monitor the applications running on cloud.gov and alert the cloud.gov team when they do not work as expected
   • Use CF security groups instead of AWS security groups and make the current rules visible to the cloud.gov team and the teams using it
   • Create a notification system to reach the key contact of each application on the platform
   • Create a more accurate staging environment to test changes like this

   View case
5. IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down

   Fastmail2016

   Symptoms

   At 4.50am Melbourne time on Sunday 7 February (evening of 6 February UTC), FastMail became unavailable and stayed unreachable for almost two hours. Traffic to FastMail's public IP range never made it into FastMail's own network.

   Service outageNetwork unreachable

   Trigger

   The GRE tunnels between Level 3 and NYI, which carried FastMail's filtered traffic into its network, went down.

   Root cause

   Dependency failure

   Component

   DDoS protection serviceBGP routing

   Contributing factor

   Limited debug access

   Detection

   NYI, the datacentre provider, noticed the failure quickly.

   Remediation

   • NYI escalated with Level 3's support until FastMail's public IP ranges were pointed back to NYI

   Action items

   • Go back to DDoS protection from Black Lotus under NYI's explicit control
   • Build nginx frontends in the Los Angeles datacentre that reach NYI over the secondary network, and point DNS to them if NYI's primary network fails
   • Add hardware in Amsterdam and change the delivery system so it depends less on NYI
   • Work with NYI to understand its processes for detecting and responding to network faults

   Lessons

   Much of FastMail's and NYI's reliability had come from NYI controlling its own infrastructure. With protection always on at Level 3, NYI could not restore the routes itself, and FastMail calls the resulting delay in restoring service unacceptable.

   View case

By root cause10