IL-0397 Zerodha · 2018-04-12networking
Orders on one exchange connection hung in pending states as its NSE leased line failed and the backup line kept flapping
Zerodha places orders on the exchanges through several CTCLs, each with a primary and a backup leased line, and orders from its trading platforms go to the CTCLs in round robin. On 12 April 2018 the primary NSE leased line of one CTCL failed, the backup line came up automatically and then flapped many times over the next hour. Orders routed to that CTCL while the lines flapped were not placed on the exchange and stayed pending. Zerodha shut the CTCL down completely so no further orders went through it, then lapsed the pending orders and reconciled them with the exchange. This affected less than 0.3% of the day's orders and about 1.5% of the clients who traded that day. For bracket and cover orders, lapsing the primary orders meant the counter leg square off orders were never placed, leaving positions that clients were advised to close by taking an opposite position using MIS/NRML. Zerodha states that it is allowed only one backup line per CTCL and has no further control over leased line reliability from the exchange-approved vendors.
- Severity
- –
- Scope
- Orders through one CTCL to NSE were not placed and hung in pending states; bracket and cover order positions were left without exit orders
- Users affected
- less than 0.3% of orders that day and around 1.5% of clients who traded that day
- Region
- India
- Time to detect
- –
- Time to resolve
- –
Symptoms
On 12 April 2018 at about 12.12 pm, the primary leased line to the NSE exchange on one of Zerodha's CTCLs went down, and the backup leased line that took over went up and down many times over the next 60 minutes. Orders sent through that CTCL were not placed on the exchange and stayed in the states open pending, validation pending and modify pending, leaving a small group of clients with hanging orders and positions.
Zerodha connects to the exchanges through several CTCLs, each with one primary and one backup leased line, and spreads all orders from its trading platforms over the CTCLs in round robin. The leased line vendor approved by NSE sent Zerodha emails when this happened. When the pending orders were lapsed, the counter leg square off orders of bracket and cover orders (BO, CO) were never placed, so those clients held BO/CO positions that could not be exited the normal way.
Symptom Link flapping
Root cause
Root cause Dependency failure
- Trigger
- The primary NSE leased line on one CTCL went down at about 12.12 pm, and the backup leased line then flapped repeatedly for about 60 minutes.
Detection
Zerodha received emails from the NSE-approved leased line vendor when the lines failed.
Remediation
- Shut the affected CTCL down completely so no new orders could go through it
- Lapsed the orders left in pending states and reconciled them with the exchange
Action items
- Release an option to exit bracket and cover orders directly from the open position, even when no counter leg open order is shown
Lessons
–
Sources
Related cases
IL-0412InfiniBand link errors and flapping links cost Meta's RSC-1 fabric as much as 50-75% of its bandwidth during bring-upMeta · 2024same root cause · shared NIC, cable and switch port link
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0397
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0397/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.