IL-0004 Grafana Labs · 2019-07-19kubernetesobservability
To upgrade without downtime, Cortex needs one extra ingester during a rollout so the old ingesters can hand their in-progress data to new ones. Each ingester requests 4 cores and 15GB of RAM, 25% of one machine, and the clusters rarely have that much free on a single machine. To make room, Grafana Labs deployed four priority classes on Thursday 18 July: critical, high, medium and low. Medium became the default for pods without a priority, ingesters were to get high, and critical was reserved for monitoring jobs. The classes had run for about a week on an internal cluster without customer traffic, but the existing production Cortex ingesters had not yet been given the high priority.
On 19 July the new customer cluster's ingesters, at the default medium priority, preempted an ingester of the production Cortex cluster, whose pods had no priority. The production ReplicaSet created a replacement ingester, which also got the default medium priority and preempted another production ingester. The cascade went on until all production ingesters had been preempted. Cortex ingesters are stateful and hold up to the last 12 hours of data. Cortex does not write to ingesters that are shutting down, so with that many of them leaving it could not replicate writes, and writes failed.
About 5 minutes after the page, the team had diagnosed the problem and scaled up the underlying Kubernetes cluster to fit both the new and the production Cortex clusters. About 5 minutes later the old ingesters had written out their data, the new ones had been scheduled, and Cortex was available again without further intervention. Fixing the out-of-memory errors in the authenticating proxies took about 10 more minutes. No data was lost: the ingesters flushed all in-memory data to long-term storage, and customers' Prometheus servers buffered failed remote writes with the new WAL-based remote write code and replayed them after the outage.
On Friday 19 July 2019 the Grafana Cloud Hosted Prometheus service, which is built on the Cortex project, was down for 26 minutes. New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start.
Writes to the production Cortex clusters failed. Ingester pods of the production clusters were being preempted one after another until none were left. Cortex replicates each series to three ingesters with quorum consistency and does not write to ingesters that are shutting down, so with many ingesters leaving its hash ring at once it could not replicate writes.
After Cortex was available again, the authenticating reverse proxies in front of it ran out of memory. Their request rate had risen more than tenfold, which Grafana Labs believes came from overly aggressive retries by customers' Prometheus servers. Diagnosing and fixing these out-of-memory errors took about another 10 minutes.
Root cause Configuration change
New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start of the outage.
Medium should not have become the default priority until the production ingesters had the high priority, and that designation should have reached production sooner. The change was considered too minor for a design document and was only discussed in a GitHub issue; cluster-global configuration needs more scrutiny. On the positive side, once Cortex had enough capacity it recovered by itself, and Grafana Loki helped confirm that the ingesters exited gracefully.
IL-0020 linked by the editorhttps://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.