Cases2 cases

1 to 2 of 2

1. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change

   UK Ministry of Justice2021

   Symptoms

   At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

   Deploys failing

   Trigger

   The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

   Root cause

   Breaking change in an upgrade

   Component

   Admission webhookIngress controller

   Contributing factor

   Unrepresentative staging

   Detection

   A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

   Remediation

   • rolled back the ModSec ingress controller module to v0.0.7

   Action items

   • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
   • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
   • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
   • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)

   View case
2. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error

   UK Ministry of Justice2020

   Symptoms

   At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.

   Deploys failing

   Trigger

   Version 0.1.0 of the team's ingress controller module started creating a ValidatingWebhookConfiguration resource, and one new 0.1.0 ingress controller in the live-1 cluster failed to be created because of AWS resource limits.

   Root cause

   Faulty deployment

   Component

   Admission webhookIngress controller

   Contributing factor

   Single point of failure

   Detection

   The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.

   Remediation

   • removed by hand all the additional admission controllers created by module version 0.1.0
   • removed the admission webhook from the ingress controller module and released version 0.1.1
   • upgraded every ingress module on version 0.1.0 to 0.1.1

   Lessons

   Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.

   View case

By root cause2