Cases2 cases

1 to 2 of 2

1. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions

   cloud.gov2016

   Symptoms

   At 7:12pm Eastern Time on 5 July 2016, logs.cloud.gov, the cloud.gov log search service, sent users into a login redirect loop, so they could not log in to the service.

   Service outageWrites rejected

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheElasticsearch or OpenSearch

   Contributing factor

   Overlapping failuresMonitoring blind spot

   Remediation

   • Moved Redis on the logsearch service to a much larger memory instance so it could keep logs while Elasticsearch was down
   • Corrected the Elasticsearch memory limits and the /tmp noexec hardening setting and redeployed the cluster
   • Redeployed the Kibana instance and verified the logs once the cluster deployment was complete

   Action items

   • Add a failure alert that checks whether logs are coming through
   • Make Redis auto-scaling or highly available so that failures in more fragile downstream components do not cause data loss
   • Extend the incident response plan to start with events that are not security-specific
   • Create a chaos bot for the logs.cloud.gov service

   View case
2. IL-0519 Apps on the cloud.gov PaaS became unavailable after an AWS security group change blocked their outbound traffic

   cloud.gov2016

   Symptoms

   From 9:28am Pacific Time on 8 March 2016, applications on the cloud.gov PaaS that relied on outbound traffic to reach the services they depended on became unavailable, and at one point every application on cloud.gov was down.

   Network unreachableService outage

   Trigger

   At 9:28am Pacific Time on 8 March 2016 a contractor at Aquilent changed the AWS security group settings, one of several changes made in the process of achieving ATO for the platform.

   Root cause

   Configuration change

   Component

   VPC route tables and firewall rules

   Contributing factor

   –

   Remediation

   • Added exceptions for individual applications, which restored some of them
   • Rolled back the security group change, which restored all applications

   Action items

   • Put all security group configuration into version control and make changes through pull requests
   • Monitor the applications running on cloud.gov and alert the cloud.gov team when they do not work as expected
   • Use CF security groups instead of AWS security groups and make the current rules visible to the cloud.gov team and the teams using it
   • Create a notification system to reach the key contact of each application on the platform
   • Create a more accurate staging environment to test changes like this

   View case

By root cause2