A collection of public post-incident reports from many companies, each read and written up in the same frame. It is for engineers who want to find incidents like the one in front of them, and for the AI tools that help those engineers.
One case is one report. Every case has the same parts: what the engineers saw, what set the incident off, the root cause, the components that broke, the factors that made it worse, what was done to end it, what the organisation decided to change afterwards, the lessons, and the address of the original report with an archived copy.
Every case is written in English. Some are also written in Korean, and those can be read in Korean.
Each case carries four kinds of category: root cause, component, symptom and contributing factor. The values come from a fixed list. Root causes include capacity exhaustion, configuration change and certificate expiry; components include CoreDNS, kubelet and the VPC CNI.
Each value has two names: the one shown on screen, and an identifier used by the API and the MCP server. The root cause shown as Certificate or credential expiry has the identifier certificate-expiry; the component CoreDNS has the identifier coredns. When you do not know an identifier, you can find it from a few words. How is described below.
The MCP server is at https://incidentlib-mcp.benchlet.com/mcp. It speaks Streamable HTTP and needs no authentication.
In Claude Code, one command is enough.
claude mcp add --transport http incidentlib https://incidentlib-mcp.benchlet.com/mcpIn Cursor, put this in the project's .cursor/mcp.json. Any other tool that takes an MCP server by address uses the same address.
{
"mcpServers": {
"incidentlib": { "url": "https://incidentlib-mcp.benchlet.com/mcp" }
}
}In Claude Desktop, open Settings, then Connectors, add a custom connector and enter the address above.
Our pods stopped resolving DNS after CoreDNS restarted. Which public incidents look like this, and what caused them?
The AI calls the tools like this and answers with case numbers and source addresses it can cite.
search_incidents(symptoms="pods cannot resolve dns after coredns restarts", limit=5)
→ interpreted: [{ kind, slug, en, ko, alias }, …]
total, results: [{ id, url, title, org, date, root_cause, symptoms, components, contributing, summary, snippet, source }, …], matched: [{ id, title, org, root_cause, … }, …], next_cursor
facets: { root_causes: [{ slug, en, ko, count, cases }], components: [...], symptoms: [...], factors: [...], orgs: [...] }
get_incident(id="IL-0001")
→ symptoms_text, trigger, root_cause, components, contributing, summary, detection, remediation, action_items, lessons, sources, similar_casessearch_incidentsget_incidentlist_incidentsget_topicvocabOne address may call the server 60 times a minute. Beyond that the answer is 429, and the Retry-After header says in how many seconds to try again. The next minute, calls are accepted again.
The base address is https://incidentlib-api.benchlet.com. Every call is a GET with no authentication and no required header. Answers are JSON with lower-case, underscored field names. Cases and category values always carry a url field with the address of the page a person would read.
curl 'https://incidentlib-api.benchlet.com/search?q=pods+cannot+resolve+dns+after+coredns+restarts&limit=3'A real answer, shortened. total is how many cases the search found. results is one page of them in full, most similar first; matched is every one of them as a row (id, title, organisation, date, root cause, symptoms, components, factors); when another page follows, next is its address. The category values your words named are in interpreted (cases carrying them rank higher), and every found case counted by category is in facets. When nothing is found, total is 0 and results and matched are empty.
{
"query": "coredns oom",
"interpreted": [
{
"kind": "component",
"slug": "coredns",
"en": "CoreDNS",
"ko": "CoreDNS",
"alias": "coredns",
"url": "https://benchlet.com/incidentlib/app/components/coredns"
},
{
"kind": "symptom",
"slug": "oom-killed",
"en": "OOMKilled",
"ko": "OOMKilled",
"alias": "oom",
"url": "https://benchlet.com/incidentlib/app/symptoms/oom-killed"
}
],
"total": 23,
"offset": 0,
"results": [
{
"id": "IL-0102",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0102",
"title": {
"en": "CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endp…",
"ko": "배포와 워커 오토스케일링 폭주가 겹치자 크기를 고정해 둔 CoreDNS 가 메모리 한도를 넘겨 재시작된 뒤 회복하지 못했고, Buildkite 사이트 전체가 약 …"
},
"org": "Buildkite",
"org_slug": "buildkite",
"date": "2026-08-25",
"root_cause": "capacity",
"contributing": [
"no-spare-capacity",
"automation-amplified",
"… 2 more"
],
"symptoms": [
"service-unavailable",
"dns-resolution-failures",
"… 2 more"
],
"components": [
"coredns",
"memory-limit",
"… 2 more"
],
"domain": [
"kubernetes",
"aws"
],
"severity": "unknown",
"summary": {
"en": "Buildkite had spent four months moving its production workloads from AWS ECS to an AWS EKS…",
"ko": "Buildkite 는 넉 달째 운영 워크로드를 AWS ECS 에서 AWS EKS 클러스터로 옮기고 있었고, 그동안 클러스터는 꾸준히 커졌다. 이 클러스터의 서비스…"
},
"match": {
"field": "action_items",
"lang": "ko",
"ord": 2,
"text": "CoreDNS 쿼리 처리 시간, 쿼리량, 고루틴 증가, 메모리 압박, OOM 재시작, 사용 가능한 레플리카 수에 직접 알림을 건다. 장기 추세도 용량 계획에 넣어…"
},
"sources": [
{
"url": "https://www.buildkitestatus.com/incidents/tm6746k61p73",
"kind": "official-status",
"license": "unknown",
"archived": null
}
]
},
{
"id": "IL-0027",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0027",
"title": {
"en": "All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole ser…",
"ko": "CoreDNS 파드가 모두 한 노드에 몰린 채 그 노드의 메모리가 고갈되자 DNS 가 응답하지 못해 아이디어스 서비스 전체가 멈췄다"
},
"org": "Backpackr",
"org_slug": "backpackr",
"date": "2025-05-07",
"root_cause": "capacity",
"contributing": [
"default-config-unchanged",
"single-point-of-failure",
"… 1 more"
],
"symptoms": [
"dns-resolution-failures",
"intermittent-connection-failures",
"… 3 more"
],
"components": [
"coredns",
"dns-resolver",
"… 3 more"
],
"domain": [
"kubernetes",
"aws",
"… 1 more"
],
"severity": "unknown",
"summary": {
"en": "Backpackr runs the idus handmade marketplace and the Tumblbug crowdfunding platform on Ama…",
"ko": "백패커는 아이디어스를 운영하는 EKS 클러스터를 처음 구성할 때 CoreDNS 를 따로 최적화하지 않고 기본 설정 그대로 두었다. CoreDNS 파드는 2개였고,…"
},
"match": {
"field": "symptoms_text",
"lang": "ko",
"ord": 3,
"text": "어느 시점에는 CoreDNS 파드가 모두 한 노드에 몰려 있었다. 그 노드의 메모리 사용률이 갑자기 치솟았고 노드에서 OOM 이 났다. kubelet 이 멈추면서…"
},
"sources": [
{
"url": "https://aws.amazon.com/ko/blogs/tech/optimizing-amazon-eks-journey-of-backpackr-part1",
"kind": "blog",
"license": "unknown",
"archived": null
}
]
}
],
"next": "https://incidentlib-api.benchlet.com/search?q=coredns+oom&limit=3&lang=en&after=cmFuazoz",
"matched": [
{
"id": "IL-0102",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0102",
"title": {
"en": "CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endp…",
"ko": "배포와 워커 오토스케일링 폭주가 겹치자 크기를 고정해 둔 CoreDNS 가 메모리 한도를 넘겨 재시작된 뒤 회복하지 못했고, Buildkite 사이트 전체가 약 …"
},
"org": "Buildkite",
"org_slug": "buildkite",
"date": "2026-08-25",
"severity": "unknown",
"root_cause": "capacity",
"symptoms": [
"service-unavailable",
"dns-resolution-failures",
"… 2 more"
],
"components": [
"coredns",
"memory-limit",
"… 2 more"
],
"contributing": [
"no-spare-capacity",
"automation-amplified",
"… 2 more"
]
},
{
"id": "IL-0027",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0027",
"title": {
"en": "All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole ser…",
"ko": "CoreDNS 파드가 모두 한 노드에 몰린 채 그 노드의 메모리가 고갈되자 DNS 가 응답하지 못해 아이디어스 서비스 전체가 멈췄다"
},
"org": "Backpackr",
"org_slug": "backpackr",
"date": "2025-05-07",
"severity": "unknown",
"root_cause": "capacity",
"symptoms": [
"dns-resolution-failures",
"intermittent-connection-failures",
"… 3 more"
],
"components": [
"coredns",
"dns-resolver",
"… 3 more"
],
"contributing": [
"default-config-unchanged",
"single-point-of-failure",
"… 1 more"
]
},
"… 3 more"
],
"facets": {
"root_causes": [
{
"slug": "capacity",
"en": "Capacity exhaustion",
"ko": "용량 한계",
"count": 15,
"cases": [
"IL-0102",
"IL-0027",
"… 13 more"
],
"url": "https://benchlet.com/incidentlib/app/root-causes/capacity"
},
{
"slug": "config-change",
"en": "Configuration change",
"ko": "설정 변경",
"count": 3,
"cases": [
"IL-0004",
"IL-0019",
"… 1 more"
],
"url": "https://benchlet.com/incidentlib/app/root-causes/config-change"
},
"… 1 more"
],
"components": [
{
"slug": "coredns",
"en": "CoreDNS",
"ko": "CoreDNS",
"count": 11,
"cases": [
"IL-0102",
"IL-0027",
"… 9 more"
],
"url": "https://benchlet.com/incidentlib/app/components/coredns"
},
{
"slug": "memory-limit",
"en": "Container memory limit",
"ko": "컨테이너 메모리 한도",
"count": 7,
"cases": [
"IL-0102",
"IL-0027",
"… 5 more"
],
"url": "https://benchlet.com/incidentlib/app/components/memory-limit"
},
"… 1 more"
],
"symptoms": [
{
"slug": "oom-killed",
"en": "OOMKilled",
"ko": "OOMKilled",
"count": 11,
"cases": [
"IL-0102",
"IL-0027",
"… 9 more"
],
"url": "https://benchlet.com/incidentlib/app/symptoms/oom-killed"
},
{
"slug": "dns-resolution-failures",
"en": "DNS resolution failures",
"ko": "DNS 조회 실패",
"count": 9,
"cases": [
"IL-0102",
"IL-0027",
"… 7 more"
],
"url": "https://benchlet.com/incidentlib/app/symptoms/dns-resolution-failures"
},
"… 1 more"
],
"factors": [
{
"slug": "default-config-unchanged",
"en": "Unreviewed defaults",
"ko": "기본 설정 방치",
"count": 7,
"cases": [
"IL-0027",
"IL-0001",
"… 5 more"
],
"url": "https://benchlet.com/incidentlib/app/factors/default-config-unchanged"
},
{
"slug": "no-spare-capacity",
"en": "No headroom",
"ko": "여유 용량 부재",
"count": 6,
"cases": [
"IL-0102",
"IL-0004",
"… 4 more"
],
"url": "https://benchlet.com/incidentlib/app/factors/no-spare-capacity"
},
"… 1 more"
],
"orgs": [
{
"slug": "uk-ministry-of-justice",
"en": "UK Ministry of Justice",
"ko": "UK Ministry of Justice",
"count": 2,
"cases": [
"IL-0019",
"IL-0020"
],
"url": "https://benchlet.com/incidentlib/app/orgs/uk-ministry-of-justice"
},
{
"slug": "zalando",
"en": "Zalando",
"ko": "Zalando",
"count": 2,
"cases": [
"IL-0001",
"IL-0031"
],
"url": "https://benchlet.com/incidentlib/app/orgs/zalando"
},
"… 1 more"
]
}
}q is the symptoms, a sentence or a few words, and is required. limit is how many cases a page holds, 5 by default and 50 at most; after takes the value from the previous page's next. lang=ko returns the Korean text where a case has one. root_cause, component, symptom, factor and domain each take a slug and keep only the cases carrying it, and org takes an organisation's slug; take the slugs from facets, interpreted or /vocab.
curl 'https://incidentlib-api.benchlet.com/cases/IL-0001'{
"id": "IL-0001",
"org": "Zalando",
"date": "2019-01-07",
"docs": [
"https://nodejs.org/docs/latest/api/dns.html#dns_implementation_considerations",
"https://github.com/kubernetes/dns/issues/159#issuecomment-336449931",
"… 4 more"
],
"title": {
"en": "Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst o…",
"ko": "CoreDNS 파드가 메모리 한도 100Mi 에 걸려 OOM 으로 계속 죽으면서 쿠버네티스 클러스터 전체의 DNS 가 멈췄다"
},
"domain": [
"kubernetes",
"aws"
],
"impact": {
"scope": "all web product and outfit pages of the Zalando Fashion Store; DNS for the entire Kubernet…",
"users": "unknown",
"region": "unknown"
},
"review": {
"written_by": "session",
"reviewed_at": null,
"reviewed_by": null
},
"lessons": {
"en": "The Kubernetes DNS setup was far from resilient enough: a single application could take do…",
"ko": "쿠버네티스 DNS 구성은 충분히 튼튼하지 않았다. 애플리케이션 하나가 DNS 를 사실상 서비스 거부 공격하듯 몰아붙여 클러스터 전체를 멈추게 할 수 있어서는 안 …"
},
"related": [
"IL-0005"
],
"sources": [
{
"url": "https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jan-2019-…",
"kind": "github",
"license": "unknown",
"archived": "http://web.archive.org/web/20261004005813/https://github.com/zalando-incubator/kubernetes-…"
}
],
"summary": {
"en": "The CoreDNS pods in the cluster had a memory limit of 100Mi, set from historical resource …",
"ko": "2019년 1월 7일 월요일, Zalando Fashion Store 의 모든 웹 상품 페이지와 코디 페이지가 한 시간 넘게 오류를 냈다. 오류는 쿠버네티스 클러…"
},
"trigger": {
"en": "Calls from the aggregation layer to a single, not very important downstream service starte…",
"ko": "그리 중요하지 않은 하위 서비스 하나로 가는 호출이 타임아웃 나기 시작했다. 집계 계층이 이 호출에 404 로 응답하자 클라이언트가 재시도했고, 집계 계층으로 가…"
},
"severity": "unknown",
"symptoms": [
"high-error-rate",
"dns-resolution-failures",
"… 2 more"
],
"timeline": {
"detected_after": null,
"resolved_after": null
},
"date_kind": "incident",
"detection": {
"en": "The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Aut…",
"ko": "DNS 장애로 클러스터 내부 모니터링이 함께 멈춰 당직 호출이 자동으로 가지 않았다. 집계 계층 서비스 당직 엔지니어가 평소처럼 자동 호출에 기대지 못하고 쿠버네…"
},
"components": [
"coredns",
"memory-limit",
"… 1 more"
],
"root_cause": "capacity",
"remediation": [
{
"text": {
"en": "Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi",
"ko": "쿠버네티스 당직 엔지니어가 CoreDNS 파드의 메모리 request 와 limit 을 100Mi 에서 2000Mi 로 직접 올렸다."
}
}
],
"action_items": [
{
"text": {
"en": "Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node",
"ko": "노드마다 dnsmasq 를 앞에 둔 CoreDNS 를 띄우는 방식으로 더 튼튼한 DNS 구성을 도입한다. 검토하고 시험한 다른 어떤 구성보다 이 방식의 성능이 좋…"
}
},
{
"text": {
"en": "Improve high-level alerting from outside the clusters, so that a cluster problem is known …",
"ko": "클러스터 바깥에서 클러스터 전체 상태를 보는 알림을 강화한다. 팀은 클러스터 안 애플리케이션이 영향을 받기 전에, 늦어도 같은 시점에는 클러스터 이상을 알아야 한…"
}
},
"… 1 more"
],
"contributing": [
"limit-unknown",
"no-caching",
"… 4 more"
],
"recurrence_of": null,
"symptoms_text": {
"en": "Customers of the Zalando Fashion Store got a high rate of errors on all web product pages …",
"ko": "Zalando Fashion Store 의 모든 웹 상품 페이지와 코디 페이지가 한 시간 넘게 고객에게 오류를 많이 냈다. 오류는 쿠버네티스 클러스터 하나에서 도…"
},
"updated_at": "2026-11-08T19:10:26.982343Z",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0001"
}curl 'https://incidentlib-api.benchlet.com/cases/IL-0001/similar'Cases with the same root cause and at least one component in common, those sharing most components first.
[
{
"id": "IL-0027",
"title": {
"en": "All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole ser…",
"ko": "CoreDNS 파드가 모두 한 노드에 몰린 채 그 노드의 메모리가 고갈되자 DNS 가 응답하지 못해 아이디어스 서비스 전체가 멈췄다"
},
"org": "Backpackr",
"date": "2025-05-07",
"severity": "unknown",
"root_cause": "capacity",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0027",
"shared_components": [
"dns-resolver",
"memory-limit",
"… 1 more"
]
},
{
"id": "IL-0048",
"title": {
"en": "DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second …",
"ko": "대형 EKS 클러스터에서 CoreDNS 가 VPC DNS 리졸버로 보내는 질의가 링크 로컬 트래픽의 초당 1024 패킷 한도에 걸려 DNS 조회가 간헐적으로 타임…"
},
"org": "unknown",
"date": "2025-08-12",
"severity": "unknown",
"root_cause": "capacity",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0048",
"shared_components": [
"coredns",
"dns-resolver"
]
},
"… 4 more"
]curl 'https://incidentlib-api.benchlet.com/cases?root_cause=capacity&limit=2'The filters are root_cause, component, symptom, factor, domain and org, and their values are identifiers. Several filters must all hold. Newest cases come first.
The list comes in pages. items holds this page's cases and, when another page follows, next holds its full address. No next means the last page. A page holds 50 cases by default and 200 at most. Use next as it is rather than building the address yourself.
{
"items": [
{
"id": "IL-0071",
"org": "Onkar Raghunath Shelke",
"date": "2026-09-17",
"docs": [
"https://dev.to/onkar_s/kubelet-watches-inodes-just-not-until-its-an-emergency-351c"
],
"title": {
"en": "Worker node running out of inodes while disk space looked fine, because containerd kept a …",
"ko": "작은 파일이 4만 개 넘게 든 단일 스테이지 이미지의 containerd 스냅샷이 쌓여 워커 노드의 inode 가 디스크보다 빠르게 소진되고 있었다"
},
"domain": [
"kubernetes",
"observability"
],
"impact": {
"scope": "one worker node trending towards inode exhaustion; no outage reported",
"users": "unknown",
"region": "unknown"
},
"review": {
"written_by": "session",
"reviewed_at": null,
"reviewed_by": null
},
"lessons": {
"en": "kubelet's background cleanup works on bytes only. A workload heavy in small files gets no …",
"ko": "kubelet 의 백그라운드 정리는 바이트를 기준으로 돈다. 용량보다 파일 수가 먼저 차는 워크로드에는 미리 손쓸 장치가 없다. kubelet 은 inode 가 …"
},
"related": [],
"sources": [
{
"url": "https://www.cncf.io/blog/2026/10/05/kubelet-watches-inodes-just-not-until-its-an-emergency…",
"kind": "blog",
"license": "cc-by",
"archived": null
}
],
"summary": {
"en": "The application image was built in one stage, so it carried the source, the build output a…",
"ko": "글쓴이가 운영하던 클러스터에서 워커 노드 한 대의 inode 가 바닥날 추세라는 알림이 왔다. 디스크 사용률은 83%, inode 사용률은 67% 였다. 알림이 …"
},
"trigger": {
"en": "Several versions of a single-stage Node.js image, each shipping more than 40,000 files, we…",
"ko": "단일 스테이지로 빌드한 이미지에 소스, 빌드 결과물, 개발 의존성까지 포함한 node_modules 전체가 들어갔고, 새 레이어 다이제스트가 생길 때마다 노드에 …"
},
"severity": "unknown",
"symptoms": [
"node-disk-full"
],
"timeline": {
"detected_after": null,
"resolved_after": null
},
"date_kind": "published",
"detection": {
"en": "The NodeFilesystemFilesFillingUp alert from the node_exporter mixin fired on the predicted…",
"ko": "node_exporter mixin 에 들어 있는 NodeFilesystemFilesFillingUp 알림이 inode 사용률 67% 에서 울렸다. predict…"
},
"components": [
"node-disk",
"kubelet",
"… 1 more"
],
"root_cause": "capacity",
"remediation": [],
"action_items": [
{
"text": {
"en": "Build the image in multiple stages so the final image contains only the compiled assets",
"ko": "멀티 스테이지 Dockerfile 로 바꿔, 한 스테이지에서 의존성을 설치하고 빌드한 뒤 최종 이미지에는 nginx:alpine 위에 컴파일된 결과물만 넣는다."
}
},
{
"text": {
"en": "Add a flat alert on root filesystem inode use above 80% for 30 minutes alongside the predi…",
"ko": "추세 알림과 별도로 루트 파일 시스템의 inode 사용률이 0.80 을 넘으면 울리는 고정 기준 알림을 for: 30m 으로 둔다."
}
},
"… 2 more"
],
"contributing": [
"default-config-unchanged",
"monitoring-blind-spot",
"… 1 more"
],
"recurrence_of": null,
"symptoms_text": {
"en": "A Kubernetes worker node paged with NodeFilesystemFilesFillingUp, the node_exporter mixin …",
"ko": "워커 노드 한 대에서 NodeFilesystemFilesFillingUp 알림이 울려 담당자가 호출됐다. inode 가 바닥날 추세라는 알림이었다.\n\n담당자는 누…"
},
"updated_at": "2026-11-08T19:10:32.156375Z",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0071"
},
{
"id": "IL-0048",
"org": "unknown",
"date": "2025-08-12",
"docs": [
"https://cep.dev/posts/eks-dns-timeouts-sudo-hostname-lookups/"
],
"title": {
"en": "DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second …",
"ko": "대형 EKS 클러스터에서 CoreDNS 가 VPC DNS 리졸버로 보내는 질의가 링크 로컬 트래픽의 초당 1024 패킷 한도에 걸려 DNS 조회가 간헐적으로 타임…"
},
"domain": [
"kubernetes",
"aws",
"… 1 more"
],
"impact": {
"scope": "DNS resolution in a very large EKS cluster, seen as intermittent CI failures",
"users": "unknown",
"region": "us-east-2, per the hostnames quoted in the post"
},
"review": {
"written_by": "session",
"reviewed_at": null,
"reviewed_by": null
},
"lessons": {
"en": "Traffic from CoreDNS to the VPC DNS resolver is limited to 1024 packets per second per int…",
"ko": "EC2 에서 링크 로컬 주소로 가는 VPC DNS 질의에는 초당 1024 패킷의 한도가 있다. 한도를 넘은 UDP 패킷은 버려지는데, 보낸 쪽은 이를 알 수 없다…"
},
"related": [],
"sources": [
{
"url": "https://medium.com/@cep21/my-process-to-debug-dns-timeouts-in-a-large-eks-cluster-9f731bd5…",
"kind": "blog",
"license": "unknown",
"archived": "https://web.archive.org/web/20250821234026/https://medium.com/@cep21/my-process-to-debug-d…"
}
],
"summary": {
"en": "Jack Lindamood describes an investigation in a very large AWS EKS cluster. Intermittent DN…",
"ko": "글쓴이의 팀은 아주 큰 AWS EKS 클러스터를 운영한다. CI 가 간헐적으로 실패한다는 보고를 따라가 보니 DNS 조회 실패였다. CoreDNS 로그에는 VPC…"
},
"trigger": {
"en": "An administrative pod running with host networking called sudo in a loop. Each sudo call l…",
"ko": "호스트 네트워크를 쓰는 시스템 관리용 파드가 몇 초마다 sudo 명령을 실행했다. sudo 는 기본 설정에서 실행할 때마다 노드 자신의 호스트 이름을 조회했다. …"
},
"severity": "unknown",
"symptoms": [
"dns-resolution-failures",
"intermittent-timeouts"
],
"timeline": {
"detected_after": null,
"resolved_after": null
},
"date_kind": "published",
"detection": {
"en": "Reports of flaky CI jobs that could not resolve an S3 bucket hostname, followed by i/o tim…",
"ko": "CI 가 간헐적으로 실패한다는 보고가 시작이었다. tensorstore 가 company-bucket.s3.amazonaws.com 의 DNS 조회에 실패했다는 …"
},
"components": [
"coredns",
"dns-resolver",
"… 1 more"
],
"root_cause": "capacity",
"remediation": [
{
"text": {
"en": "added 'Defaults !fqdn' in /etc/sudoers.d/00-no-fqdn (mode 440) to the container image so s…",
"ko": "컨테이너 이미지에 /etc/sudoers.d/00-no-fqdn 파일로 Defaults !fqdn 을 넣어 sudo 가 호스트 이름을 조회하지 않게 했다."
}
}
],
"action_items": [],
"contributing": [
"limit-unknown",
"default-config-unchanged"
],
"recurrence_of": null,
"symptoms_text": {
"en": "CI jobs failed now and then because a hostname could not be resolved. The failures came fr…",
"ko": "처음 들어온 보고는 CI 가 간헐적으로 실패한다는 것이었다. 실패는 tensorstore 안에서 났고, 오류는 ./tensorstore/util/result.h:…"
},
"updated_at": "2026-11-08T19:10:30.194994Z",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0048"
}
],
"next": "https://incidentlib-api.benchlet.com/cases?root_cause=capacity&limit=2&after=MjAyNS0wOC0xM…"
}curl 'https://incidentlib-api.benchlet.com/symptoms/dns-resolution-failures'Root causes are at /root-causes/identifier, components at /components/identifier, symptoms at /symptoms/identifier factors at /factors/identifier and organisations at /orgs/identifier. All five answer in the same shape. Without an identifier, each lists every value of that category with its case count. Below is the symptom DNS resolution failures; related.root_causes is which root causes produced it and how often.
{
"kind": "symptom",
"slug": "dns-resolution-failures",
"en": "DNS resolution failures",
"ko": "DNS 조회 실패",
"url": "https://benchlet.com/incidentlib/app/symptoms/dns-resolution-failures",
"count": 11,
"first_date": "2018-07-19",
"last_date": "2025-08-12",
"cases": [
{
"id": "IL-0048",
"title": {
"en": "DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second …",
"ko": "대형 EKS 클러스터에서 CoreDNS 가 VPC DNS 리졸버로 보내는 질의가 링크 로컬 트래픽의 초당 1024 패킷 한도에 걸려 DNS 조회가 간헐적으로 타임…"
},
"org": "unknown",
"date": "2025-08-12",
"severity": "unknown",
"root_cause": "capacity",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0048"
},
{
"id": "IL-0027",
"title": {
"en": "All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole ser…",
"ko": "CoreDNS 파드가 모두 한 노드에 몰린 채 그 노드의 메모리가 고갈되자 DNS 가 응답하지 못해 아이디어스 서비스 전체가 멈췄다"
},
"org": "Backpackr",
"date": "2025-05-07",
"severity": "unknown",
"root_cause": "capacity",
"url": "https://benchlet.com/incidentlib/app/cases/IL-0027"
},
"… 9 more"
],
"related": {
"root_causes": [
{
"slug": "capacity",
"en": "Capacity exhaustion",
"ko": "용량 한계",
"count": 5,
"cases": [
"IL-0075",
"IL-0048",
"… 3 more"
],
"url": "https://benchlet.com/incidentlib/app/root-causes/capacity"
},
{
"slug": "dns-network",
"en": "DNS or network fault",
"ko": "DNS·네트워크 장애",
"count": 2,
"cases": [
"IL-0049",
"IL-0047"
],
"url": "https://benchlet.com/incidentlib/app/root-causes/dns-network"
},
"… 3 more"
],
"components": [
{
"slug": "dns-resolver",
"en": "Client-side DNS resolver and cache",
"ko": "클라이언트 쪽 DNS 리졸버와 캐시",
"count": 7,
"cases": [
"IL-0050",
"IL-0048",
"… 5 more"
],
"url": "https://benchlet.com/incidentlib/app/components/dns-resolver"
},
{
"slug": "coredns",
"en": "CoreDNS",
"ko": "CoreDNS",
"count": 4,
"cases": [
"IL-0049",
"IL-0048",
"… 2 more"
],
"url": "https://benchlet.com/incidentlib/app/components/coredns"
},
"… 13 more"
],
"factors": [
{
"slug": "default-config-unchanged",
"en": "Unreviewed defaults",
"ko": "기본 설정 방치",
"count": 6,
"cases": [
"IL-0050",
"IL-0048",
"… 4 more"
],
"url": "https://benchlet.com/incidentlib/app/factors/default-config-unchanged"
},
{
"slug": "monitoring-blind-spot",
"en": "Monitoring blind spot",
"ko": "모니터링 사각지대",
"count": 4,
"cases": [
"IL-0075",
"IL-0060",
"… 2 more"
],
"url": "https://benchlet.com/incidentlib/app/factors/monitoring-blind-spot"
},
"… 16 more"
]
}
}With a symptom sentence in q, only the similar cases stay and the related categories are counted over those cases only.
curl 'https://incidentlib-api.benchlet.com/root-causes/capacity?q=dns+lookups+fail'curl 'https://incidentlib-api.benchlet.com/vocab?q=cert+ran+out&kind=root_causes&limit=3'Words in q return the category values closest in meaning, closest first. kind narrows to one category and takes root_causes, components, symptoms, factors or orgs. Without q, every value comes back with its English and Korean names.
{
"matches": [
{
"kind": "root_cause",
"slug": "certificate-expiry",
"en": "Certificate or credential expiry",
"ko": "인증서·자격 증명 만료",
"url": "https://benchlet.com/incidentlib/app/root-causes/certificate-expiry"
},
{
"kind": "root_cause",
"slug": "credential-exposure",
"en": "Credential exposure",
"ko": "자격 증명 유출",
"url": "https://benchlet.com/incidentlib/app/root-causes/credential-exposure"
},
"… 1 more"
],
"query": "cert ran out"
}id | The case number, such as IL-0001. It never changes. |
url | The page where a person reads this case. |
title | The title, in English and in Korean. |
org | The organisation that had the incident. |
date | The day of the incident, or the day the report was published. |
severity | sev1 to sev4; unknown when the report does not say. |
domain | kubernetes or ai-infra. |
root_cause | The identifier of the root cause category. |
components | The identifiers of the components that broke. |
symptoms | The identifiers of the symptom categories. |
contributing | The identifiers of the contributing factor categories. |
symptoms_text | What the engineers saw, as sentences. |
trigger | What set the incident off. |
summary | What happened and how, in a few sentences. |
detection | How it was noticed. |
remediation | What was done to end the incident. |
action_items | What the organisation decided to change afterwards. |
lessons | The lessons the report draws. |
sources | The original address, the archived copy, the kind of source and its licence. |
impact | Scope, users affected and region. |
timeline | Time to detect and time to resolve. |
recurrence_of | The number of an earlier case of the same incident, when there is one. |
related | Case numbers an editor linked as related. |
updated_at | When this case last changed. |
A text field that exists in both languages comes as two entries, en and ko. A case without Korean has en only.
Every error has the same shape. The content type is application/problem+json; status is the status code, title its name, detail one sentence on what went wrong, and instance the path that was called. Asking for a case that does not exist gives this.
curl -i 'https://incidentlib-api.benchlet.com/cases/IL-9999'{
"type": "about:blank",
"title": "Not Found",
"status": 404,
"detail": "no approved case IL-9999",
"instance": "/cases/IL-9999"
}400 | A parameter is missing or out of range. detail says which. |
404 | No case with that number, or no category value with that identifier. |
429 | This address went over its allowance for the minute. Retry-After says in how many seconds to try again. |
503 | Try again in a moment. |
500 | Something unexpected failed on the server. Try again in a moment; if it keeps happening, tell us at the address below. |
One address may make 30 searches and 120 other calls a minute. Beyond the allowance the answer is 429, and the next minute calls are accepted again. If you need more, tell us at the address below.
| Request | No header is required. |
Content-Type | application/json; charset=utf-8 on a normal answer, application/problem+json on an error. |
Access-Control-Allow-Origin | Always *: browser code on any site may read this API. |
Retry-After | Only on 429, the number of seconds to wait. |
The machine-readable description is the OpenAPI document.
The cases say what went wrong elsewhere. kubegap looks for the same conditions in a Kubernetes cluster of yours. It is an Agent Skill, a folder of procedure and scripts that Claude Code or Codex follows: type /kubegap and the agent reads the cluster through kubectl, evaluates it against several hundred checks and writes a report with the evidence, the cause and a suggested fix for each problem. It only reads, installs nothing in the cluster and keeps the collected data on your machine.
Many of its checks cite the public reports this library holds, so a case found here and a finding there often point at the same report. The install command and the full list of checks are on its page.
Every case comes from a public post-incident report: a company engineering blog, a status page, a talk, a paper or a code repository. The case text is a summary of what the report says, in our own words, and a fact the report does not state is left empty. Each source address has an archived copy where one could be made.
The case texts, the category list and the API answers are available under CC BY 4.0. The original reports keep their own licences, named in each case's sources. To report a mistake or propose a case, open an issue on GitHub.