Incident Library

Organisation

Chatwork

Chatwork · chatwork

1 case · 2021

Root cause Capacity exhaustion
Component conntrack tableCoreDNSClient-side DNS resolver and cachekube-proxyVPC DNS resolver
Contributing factor Unreviewed defaultsUnknown limitUnrepresentative staging
Symptom DNS resolution failuresIntermittent connection failuresIntermittent timeoutsLoad imbalance

Cases

  1. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were droppedChatwork · 2021

    Root cause Capacity exhaustion