Incident Library
Applied filters ComponentTelemetry agentSymptomLog floodOrganisationGoogle CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customers
    Google Cloud2023
    Symptoms

    From 07:13 US/Pacific on 6 July 2023, GKE customers saw unexpected additional error and warning entries in their cluster logs, written to the Cloud Logging project that holds the clusters. Standard and Autopilot clusters at GKE 1.24.x and later were affected.

    Log flood
    Trigger

    Google rolled out a new version of the GKE component that processes logs (preliminary analysis).

    Root cause
    Faulty deployment
    Component
    Telemetry agent
    Contributing factor
    –
    Remediation
    • gave customers a workaround: request a Logging API quota increase, or apply a log exclusion for the fluentbit-gke messages
    • rolled out a fix gradually to the affected clusters between 10 and 18 July 2023
    Lessons

    Log messages written by a managed logging agent count against the customer's own Logging API write quota and logging bill, so a faulty agent release can drop a customer's other logs. A log exclusion on the agent's container name and the message patterns keeps the noise out of storage until a fix arrives.

    View case
Check your cluster for this

This case involves Telemetry agent. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Telemetry agent
Contributing factor
–
Organisation
Google Cloud
  1. IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customers
    Google Cloud2023