Cases4 cases

1 to 4 of 4

1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed

   Railway2026

   Symptoms

   On 30 September 2026, between about 07:35 and 07:40 UTC, new connections to every domain hosted on Railway, both custom domains and *.up.railway.app domains, got HTTP 404 in all regions. Internal alerting paged the on-call team as soon as route lookups started to fail.

   Service outageDNS resolution failures

   Trigger

   At 07:29 UTC an engineer merged a change to the routing control plane that contained a database schema update and a new version of the routing service. At about 07:35 the deployment system rolled the new routing service out to every region before the schema update had been applied.

   Root cause

   Faulty deployment

   Component

   Network control plane (SDN)DatabaseCI system

   Contributing factor

   Unsafe failure behaviourNo staged rolloutExcessive blast radius

   Detection

   Internal alerting engaged the on-call response team as soon as route lookups began to fail at about 07:35 UTC.

   Remediation

   • Let the CI system finish the schema update, after which the routing service's retried lookups succeeded and routing recovered without a rollback

   Action items

   • Force the migration rollout to run in serial order and unify the deployment pipeline so deploy order and blast radius are checked before merge
   • Block the routing service's blue/green rollout until the database schema matching its version is in place
   • Add shard based rollouts per region to the distributed control plane to isolate impact

   View case
2. IL-0268 AdGuard DNS resolvers in Amsterdam, Frankfurt and London overloaded and answered with timeouts and SERVFAIL

   AdGuard DNS2025

   Symptoms

   Between about 15:30 and 18:00 UTC on 25 November 2025 some AdGuard DNS users saw slow or failed DNS resolution, with timeouts and SERVFAIL responses. Alerts fired for increased DNS error rates and server load, and a significant part of the traffic in several European locations, primarily Amsterdam, Frankfurt and London, had intermittent DNS failures for up to 2.5 hours.

   DNS resolution failuresIntermittent timeoutsHigh latencyMemory pressure

   Trigger

   A new version of AdGuard DNS, deployed on 25 November to improve how resolvers cache user data, had a bug that saved only the recently changed subset of user data into the local cache file. The first deployment failed partway on a network issue and left about half the servers on the new version; the second attempt made many resolvers start full synchronizations at once.

   Root cause

   Faulty deployment

   Component

   Recursive DNS resolver serviceLanguage runtime settings

   Contributing factor

   Dependence on a warm cacheRetry stormHarmful manual actionGap in automated tests

   Detection

   Alerts for increased DNS error rates and server load fired during the second deployment attempt, between 15:30 and 16:00 UTC.

   Remediation

   • Rolled back the buggy version on all servers
   • Partially rerouted Amsterdam's traffic to other European locations
   • Reduced connection concurrency and request pipeline depth in Frankfurt and London, from 5 to 1 in Frankfurt
   • Adjusted limits to the actual hardware capacity
   • Repaired corrupted cache files and user data snapshots to stop further full synchronizations
   • Normalised the traffic distribution once the rollback was complete

   Action items

   • Add automated tests that the full user dataset is correctly written to and restored from the local cache file, including detection of empty or truncated caches
   • Add integration tests that simulate resolver startup with an empty or corrupted cache under realistic load
   • Standardize server hardware profiles per location, or keep separate configurations per hardware class
   • Review and adjust connlimit, cache sizes and connection limits for each standard node type
   • Reevaluate and lower pipelining depth where needed
   • Reevaluate how connections and goroutines are limited so resource use stays bounded with decent throughput after a connection break
   • Monitor Go heap size, GC time, GC cycles per second and per-cluster request latency distributions
   • Alert on sustained increases in SERVFAIL rate and request duration while upstreams are healthy
   • Refine DNS resolver deployment and rollback procedures to avoid simultaneous cache invalidation across a large part of the fleet
   • Document and rehearse playbooks for adjusting pipeline depth, connection limits and connlimit during live incidents

   Lessons

   How nodes behaved depended on the interaction between available RAM, Go's garbage collector and the connlimit setting. On 64-GiB nodes a high limit let the heap grow large under heavy concurrency and pipelining, and the garbage collector then spent more and more CPU, a known worst case for high-memory applications in which memory is not exhausted but latency rises and throughput falls. On 32-GiB nodes the same configuration reached practical memory limits sooner because it had not been scaled down. Together with high pipelining and client retries, GC pressure, connection saturation and retries formed a feedback loop.

   View case
3. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN

   bunny.net2021

   Symptoms

   On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

   Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance

   Trigger

   At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

   Root cause

   Faulty deployment

   Component

   Authoritative DNS servers and recordsCDN and edge networkObject storage

   Contributing factor

   Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action

   Detection

   About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

   Remediation

   • Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
   • Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
   • Migrated the deployment systems and files to a third-party cloud storage service
   • Rewired the deployment system and DNS software to the new storage and deployed
   • Synced the databases and deployed the newest file sets through the third-party service

   Action items

   • Phase out the BinaryPack library as a hot-fix
   • Test third-party libraries more extensively
   • Migrate the internal APIs to an independent third-party service
   • Find how to stop non-critical software from becoming a single point of failure across clusters
   • Make the DNS system keep a local copy of all backup data with automatic failure detection

   Lessons

   bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

   View case
4. IL-0360 DNS queries to bunny.net timed out after a failed routing update took two of four nameserver clusters offline

   bunny.net2021

   Symptoms

   On 26 May 2021, after routine maintenance at about 20:40 UTC, bunny.net saw an abnormal level of load on one of its DNS clusters, NS cluster 1, and DNS queries started to time out. About 10% of traffic was affected for roughly 30 minutes, although bunny.net runs three independent anycast networks. The timeouts made resolvers send even more queries, and that extra load gradually overloaded one of the nameserver clusters.

   DNS resolution failures

   Trigger

   During the maintenance, bunny.net deployed a routing update to its DNS system meant to improve performance and reliability, and an issue with a different system made the update deploy incorrectly.

   Root cause

   Faulty deployment

   Component

   Authoritative DNS servers and records

   Contributing factor

   Retry stormNo headroom

   Detection

   bunny.net detected an abnormal level of load on one of its DNS clusters after the maintenance.

   Action items

   • Add quality control and monitoring steps so a failed DNS routing update does not happen again
   • Install additional capacity in all nameserver clusters so that each one can handle the full load
   • Work with the team so the human error behind the incident does not happen again

   View case

By root cause1