Cases3 cases

1 to 3 of 3

1. IL-0558 New PubNub instance types in FRA hit connectivity limits, and Pub/Sub, History and Presence saw intermittent errors

   PubNub2025

   Symptoms

   From 08:15 UTC on Friday 27 June 2025, PubNub's Pub/Sub, History and Presence services showed occasional, intermittent increases in latency and server errors in the Europe (FRA) region.

   High latencyHigh error rate

   Trigger

   PubNub introduced new instance types to its system to get access to more cloud server capacity across its regions.

   Root cause

   Capacity exhaustion

   Component

   Cloud load balancer

   Contributing factor

   Partial rolloutUnverified assumptionMonitoring blind spot

   Remediation

   • Configured the affected services to run on other instance types
   • Launched more capacity

   Action items

   • Make the new instance types emit metrics on OS thresholds and limits, so approaching or exceeding a limit is detected on any instance type
   • Scale ahead of demand and route traffic according to instance type

   View case
2. IL-0578 Static content behind an HAProxy load balancer in an OpenStack Octavia amphora was delayed by tens of seconds

   GMO Pepabo2020

   Symptoms

   In a GMO Pepabo production setup, delivery of static content was delayed by tens of seconds. The static content was served by nginx on upstream hosts behind HAProxy, which ran as the HTTP load balancer.

   High latency

   Root cause

   Capacity exhaustion

   Component

   HAProxyLinux kernelCloud load balancer

   Contributing factor

   Unreviewed defaults

   Remediation

   • Tuned net.ipv4.tcp_mem with sysctl on the HAProxy host, which ended the delays immediately

   Lessons

   On a VM with little RAM, the boot-time default of net.ipv4.tcp_mem can be low enough to push TCP into memory pressure. TCP out of memory is not the OOM killer, and net.ipv4.tcp_mem is a system-wide limit counted in pages, unlike the per-socket net.ipv4.tcp_rmem and net.ipv4.tcp_wmem counted in bytes. The values to watch are TCP mem in /proc/net/sockstat, press in /proc/net/protocols, and TCPMemoryPressures, TCPMemoryPressuresChrono and TCPAbortOnMemory in /proc/net/netstat.

   View case
3. IL-0333 A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com down

   Buildkite2016

   Symptoms

   On 22 August 2016 at 17:21 UTC buildkite.com went offline: nobody could log in, view build logs or read the documentation, although builds kept running and GitHub and Bitbucket pull request statuses kept updating. The Buildkite team in Melbourne was asleep and woke at 21:00 UTC, almost 4 hours later, to emails, tweets and Slack messages saying Buildkite was down.

   Service outageDatabase overloadProbe failuresCloud API rate limit exhausted

   Trigger

   Over a weekend maintenance window, Buildkite downgraded its m4.10xlarge Multi-AZ RDS PostgreSQL database to an r3.2xlarge Multi-AZ instance to cut its AWS bill, and moved to a centralised pgbouncer to pool connections. The first US weekday peak after the change pushed the new instance's CPU to its maximum at about 16:00 UTC.

   Root cause

   Capacity exhaustion

   Component

   DatabaseCloud load balancerLiveness and readiness probesNode launch template

   Contributing factor

   Unverified assumptionHealth check tied to a dependencyAutomation amplificationCircular dependencyPages not deliveredOverlapping failures

   Detection

   New Relic, Pingdom, Datadog, Calibre and CloudWatch raised PagerDuty alerts, but the on-call phone call never came and only silent push notifications arrived. The team noticed at 21:00 UTC when they woke to emails, tweets and Slack messages from users.

   Remediation

   • Moved servers from the Agent API to the dashboard ELB to bring buildkite.com back online
   • Switched the Auto Scaling Groups to EC2 health checks and locked in desired counts
   • Upgraded the database to an instance type that could handle the load, using the aws-cli
   • Built a new AMI that did not depend on buildkite.com/_secret/version and rolled out servers that could self-deploy
   • Removed the centralised pgbouncer and went back to pgbouncer on each host

   Action items

   • Plan load testing after significant infrastructure changes without affecting other users of the system
   • Make the ELB health check return OK when the app answers, without checking database connectivity
   • Consider a separate health check endpoint for Pingdom that checks the database and other dependencies
   • Store the latest deployed revision as a file on S3 for newly launched servers to self-deploy from
   • Test by hand that PagerDuty calls reach every on-call team member
   • Investigate regular company-wide tests of everyone's paging setup
   • Ask PagerDuty about calling members when they go on call or requiring an immediate phone rule

   Lessons

   Buildkite listed six lessons. Watch AWS credits, because billing reports show $0 while credits are spent, so the real cost is hidden and big changes end up rushed. Load test after significant infrastructure changes. Think about what a health check should do and what happens when every server behind an ELB fails it. Do not make new servers look up the deployed revision from the service they are replacing. Make sure every on-call member has the right PagerDuty and phone settings before going on call. When servers churn in AWS, stop health checks and autoscaling, for example with the Auto Scaling feature to suspend and resume processes, so the EC2 request limit is not used up.

   View case

By root cause1