How to Set Up Fluent Bit on Kubernetes: Helm, Loki, OpenSearch, S3 (2026)

This guide is for platform engineers who need container logs from every Kubernetes node delivered somewhere useful, without losing them when the backend has a bad afternoon. At the end you will have Fluent Bit 5.1.3 running as a DaemonSet from the official Helm chart 0.58.3, configured in YAML rather than the classic format, parsing CRI logs and Java or Python stack traces, enriching each record with pod metadata, buffering to disk, and shipping to Loki, OpenSearch and S3. I rendered the chart with these exact values using helm template (Helm 3.14.4) on 24 October 2026 and checked the options against the 5.1 docs and source. I did not run it on a live cluster for this post, so the cluster-side output below is abridged from the documented formats and the 5.1.3 source strings. Prerequisites: a cluster with containerd nodes, Helm 3.8 or later (for OCI charts), kubectl, and at least one log backend.

Step 1: Know what Fluent Bit reads and which version to pin

The kubelet and the container runtime write each container's stdout and stderr to files on the node in the CRI logging format, and the kubelet rotates them: by default at 10 MiB per file, keeping 5 files per container. Only the newest file is visible to kubectl logs, which is why you need an agent. The files that agents read are the symlinks in /var/log/containers/, named <pod>_<namespace>_<container>-<container-id>.log, and the kubernetes filter parses pod, namespace and container out of that name.

Pin 5.1. Fluent Bit 5.0.0 was announced on 24 March 2026, 5.1.0 followed in August, and the latest release is v5.1.5 (21 October 2026). The chart's appVersion is 5.1.3, so that is what this guide pins; bump the image tag when the chart catches up. The 4.2 line last shipped v4.2.8 on 4 August 2026. If you are upgrading from 4.x, read the upgrade notes: fluentbit_hot_reloaded_times became a counter, and from 5.0.4 the forward output sets retain_metadata_in_forward_mode: true by default, which Fluentd receivers reject.

Use the YAML format. The Fluent Bit docs say plainly that classic configuration files will be deprecated at the end of 2026, and YAML supports features classic files do not, such as processors. The chart still renders a classic fluent-bit.conf by default, so the values below add a YAML file and point the binary at it.

Step 2: Write the values file

Save this as values.yaml. It replaces the chart's default host mounts (the default also mounts /var/lib/docker/containers, which containerd nodes do not have), adds a persistent state directory for the tail database and buffers, and writes the whole Fluent Bit config as fluent-bit.yaml. The output is stdout for now, so you can see records before wiring a backend.

# values.yaml for oci://ghcr.io/fluent/helm-charts/fluent-bit 0.58.3 (Fluent Bit 5.1.3)
image:
  tag: "5.1.3"            # pin the binary; the chart default is its appVersion, also 5.1.3

# Run the YAML config instead of the chart's classic fluent-bit.conf
args:
  - --workdir=/fluent-bit/etc
  - --config=/fluent-bit/etc/conf/fluent-bit.yaml

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 512Mi

# Tolerate every taint so control-plane and GPU nodes ship logs too
tolerations:
  - operator: Exists

daemonSetVolumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: etcmachineid
    hostPath:
      path: /etc/machine-id
      type: File
  - name: state                      # tail offsets and buffers survive restarts
    hostPath:
      path: /var/fluent-bit/state
      type: DirectoryOrCreate

daemonSetVolumeMounts:
  - name: varlog
    mountPath: /var/log
    readOnly: true
  - name: etcmachineid
    mountPath: /etc/machine-id
    readOnly: true
  - name: state
    mountPath: /var/fluent-bit/state

config:
  extraFiles:
    fluent-bit.yaml: |
      service:
        flush: 1
        log_level: info
        http_server: on
        http_listen: 0.0.0.0
        http_port: 2020
        health_check: on              # the chart's readiness probe hits /api/v2/health
        parsers_file: /fluent-bit/etc/parsers.conf
        storage.path: /var/fluent-bit/state/flb-storage/
        storage.sync: normal
        storage.checksum: off
        storage.max_chunks_up: 128
        storage.backlog.mem_limit: 5M
        storage.metrics: on

      pipeline:
        inputs:
          - name: tail
            tag: kube.*
            path: /var/log/containers/*.log
            exclude_path: /var/log/containers/fluent-bit-*_logging_*.log   # never tail yourself
            multiline.parser: docker, cri
            db: /var/fluent-bit/state/tail-kube.db
            db.locking: true
            refresh_interval: 10        # default 60s misses short-lived Job pods
            rotate_wait: 30
            skip_long_lines: on
            storage.type: filesystem

        filters:
          - name: multiline             # must be the first filter
            match: kube.*
            multiline.key_content: log
            multiline.parser: java, python, go

          - name: kubernetes
            match: kube.*
            merge_log: on               # parse JSON in "log" into top-level fields
            keep_log: off
            k8s-logging.parser: on
            k8s-logging.exclude: on
            labels: on
            annotations: off
            buffer_size: 0              # no limit; large pod specs overflow the 32k default

        outputs:
          - name: stdout
            match: kube.*
            format: json_lines

A few choices are worth explaining. The tag kube.* expands to kube.var.log.containers.<file>, which matches the kubernetes filter's default kube_tag_prefix, so no prefix override is needed. multiline.parser: docker, cri rejoins lines the runtime split, trying docker first, then cri, per the tail input docs. The multiline filter handles application stack traces; its docs say it should be the first filter, because it re-emits records and any filter before it would run twice. Newly discovered files are read from the start by default (read_newly_discovered_files_from_head), so the low refresh_interval is what catches pods that live for seconds.

Step 3: Install with Helm and verify

The chart README recommends the OCI registry and signs releases with Sigstore. Verifying first is optional and takes one command.

# Optional: verify the chart signature (command from the chart README)
cosign verify \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp 'https://github\.com/action-stars/helm-workflows/\.github/workflows/release\.yaml@.+' \
  --certificate-github-workflow-repository fluent/helm-charts \
  --certificate-github-workflow-name Release \
  ghcr.io/fluent/helm-charts/fluent-bit:0.58.3

helm upgrade --install fluent-bit oci://ghcr.io/fluent/helm-charts/fluent-bit \
  --version 0.58.3 --namespace logging --create-namespace -f values.yaml

kubectl -n logging rollout status ds/fluent-bit
kubectl -n logging logs ds/fluent-bit | grep -E "version=|\[storage\]|connectivity"

Expected lines (timestamps trimmed; the strings come from the 5.1.3 source):

[ info] [fluent bit] version=5.1.3, commit=..., pid=1
[ info] [storage] ver=..., type=memory+filesystem, sync=normal, checksum=off, max_chunks_up=128
[ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...
[ info] [filter:kubernetes:kubernetes.1] connectivity OK

kubernetes.1 is correct: filter instance numbers are global, and the multiline filter is instance 0. Now check the HTTP API and counters:

kubectl -n logging port-forward ds/fluent-bit 2020:2020 &
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:2020/api/v2/health   # 200 healthy, 500 unhealthy
curl -s http://127.0.0.1:2020/api/v2/metrics/prometheus \
  | grep -E 'fluentbit_(input_records|output_proc_records|output_errors|output_retries_failed|output_dropped_records)_total'

Step 4: Test JSON parsing and stack traces

Run a pod that prints one JSON line and one three-line Java exception, then read the Fluent Bit pod on the same node.

kubectl run log-demo --image=busybox:1.38.0 --restart=Never -- sh -c '
  echo "{\"level\":\"info\",\"msg\":\"order placed\",\"order_id\":42}"
  printf "java.lang.IllegalStateException: boom\n\tat com.example.App.run(App.java:12)\n\tat com.example.App.main(App.java:5)\n"
  sleep 3600'

NODE=$(kubectl get pod log-demo -o jsonpath='{.spec.nodeName}')
FB=$(kubectl -n logging get pods -l app.kubernetes.io/name=fluent-bit \
  --field-selector spec.nodeName="$NODE" -o name)
kubectl -n logging logs "$FB" | grep log-demo

You should see two records, abridged here:

{"date":...,"stream":"stdout",...,"level":"info","msg":"order placed","order_id":42,"kubernetes":{"pod_name":"log-demo","namespace_name":"default",...}}
{"date":...,"stream":"stdout",...,"log":"java.lang.IllegalStateException: boom\n\tat com.example.App.run(App.java:12)\n\tat com.example.App.main(App.java:5)...","kubernetes":{...}}

The JSON line was merged into top-level fields because of merge_log: on, and the exception arrived as one record, not three, because the built-in java parser matched the ...Exception: first line and the indented at lines. If your apps already log JSON, as they should (see my structured logging best practices), most of the multiline work disappears.

Pods can opt out or choose a parser through annotations in their pod template, which the filter honours because k8s-logging.exclude and k8s-logging.parser are on. The exclude value must be the quoted string "true":

metadata:
  annotations:
    fluentbit.io/exclude: "true"     # drop this pod's logs
    # fluentbit.io/parser: nginx     # or apply a parser from parsers.conf

Set these in the pod template, not on a running pod: with kube_meta_cache_ttl at its default of 0, cached metadata never expires.

Step 5: Send logs to Loki

Replace the stdout output with the Loki block. Keep labels low-cardinality (namespace and container) and put the pod and node in structured metadata, which the Loki output docs recommend for high-cardinality data. The host matches the gateway from my Loki on Kubernetes guide.

        outputs:
          - name: loki
            match: kube.*
            host: loki-gateway.loki.svc.cluster.local
            port: 80
            labels: job=fluent-bit, namespace=$kubernetes['namespace_name'], container=$kubernetes['container_name']
            structured_metadata: pod=$kubernetes['pod_name'], node=$kubernetes['host']
            line_format: json
            retry_limit: false               # retry forever; disk use is capped below
            storage.total_limit_size: 2G     # oldest chunks for this output dropped past 2G

helm upgrade fluent-bit oci://ghcr.io/fluent/helm-charts/fluent-bit \
  --version 0.58.3 --namespace logging -f values.yaml

In Grafana Explore, {job="fluent-bit", namespace="default", container="log-demo"} | json should return both test records.

Step 6: Send logs to OpenSearch

Store credentials in a Secret, expose them as environment variables, and mount the cluster CA. Fluent Bit substitutes ${VAR} in config values.

kubectl -n logging create secret generic opensearch-credentials \
  --from-literal=username=fluent-bit --from-literal=password='change-me'
kubectl -n logging create secret generic opensearch-ca --from-file=ca.crt=./opensearch-ca.crt

Add to values.yaml at the top level:

env:
  - name: OPENSEARCH_USER
    valueFrom:
      secretKeyRef: {name: opensearch-credentials, key: username}
  - name: OPENSEARCH_PASSWORD
    valueFrom:
      secretKeyRef: {name: opensearch-credentials, key: password}

extraVolumes:
  - name: opensearch-ca
    secret:
      secretName: opensearch-ca
extraVolumeMounts:
  - name: opensearch-ca
    mountPath: /fluent-bit/etc/opensearch-ca
    readOnly: true

And this output to pipeline.outputs:

          - name: opensearch
            match: kube.*
            host: opensearch-cluster-master.opensearch.svc.cluster.local
            port: 9200
            http_user: ${OPENSEARCH_USER}
            http_passwd: ${OPENSEARCH_PASSWORD}
            tls: on
            tls.verify: on
            tls.ca_file: /fluent-bit/etc/opensearch-ca/ca.crt
            logstash_format: on              # daily indices: kube-YYYY.MM.DD
            logstash_prefix: kube
            suppress_type_name: on           # required for OpenSearch 2.0 and later
            replace_dots: on                 # app vs app.kubernetes.io/name mapping clash
            trace_error: on                  # print the response body when a bulk request fails
            retry_limit: false
            storage.total_limit_size: 2G

Per the OpenSearch output docs, OpenSearch 2.0 and later reject the _type field unless suppress_type_name is on. For Amazon OpenSearch Serverless, set aws_auth: on, aws_region, and aws_service_name: aoss instead of a user and password. Set retention with an Index State Management policy on the kube-* indices; Fluent Bit does not delete anything.

Step 7: Archive logs to S3

Grant the service account s3:PutObject on the bucket. On EKS, either create a Pod Identity association (it needs the Pod Identity Agent add-on and a role trusting pods.eks.amazonaws.com with sts:AssumeRole and sts:TagSession) or annotate the service account for IRSA. Fluent Bit's AWS credential chain supports both.

aws eks create-pod-identity-association --cluster-name my-cluster \
  --role-arn arn:aws:iam::111122223333:role/fluent-bit-s3 \
  --namespace logging --service-account fluent-bit
kubectl -n logging rollout restart ds/fluent-bit     # credentials are injected at pod start

          - name: s3
            match: kube.*
            bucket: my-cluster-log-archive
            region: eu-west-1
            total_file_size: 100M
            upload_timeout: 10m
            store_dir: /var/fluent-bit/state/s3          # on the hostPath, so restarts resume uploads
            s3_key_format: /kubernetes/%Y/%m/%d/%H/$UUID.json.gz
            compression: gzip
            retry_limit: 5
            storage.total_limit_size: 5G

The S3 output docs say to always include $UUID in the key, and the extension is not added for you, hence .json.gz. One thing the docs do not spell out: the plugin keeps a separate buffer file per tag (I checked s3_store.c in 5.1.3), and with kube.* every container has its own tag, so expect one object per container per upload window. If that is too many small objects for Athena, re-tag per namespace with the rewrite_tag filter before the S3 output. Version 5.1 also added format: parquet, which requires use_put_object: on.

Step 8: Configure buffering and backpressure

Fluent Bit has two buffering modes, and the difference decides whether an outage loses logs. With memory buffering, mem_buf_limit caps each input; when it is hit, the input is paused and logs paused (mem buf overlimit). A paused tail stops reading, and if the kubelet rotates past its five files before Fluent Bit resumes, those lines are gone.

With storage.type: filesystem, which this guide uses, chunks of about 2 MB are written to storage.path and only up to storage.max_chunks_up (128) are held in memory, so 128 chunks is roughly 256 MB; keep the container memory limit above that or lower the number. The input keeps writing to disk while an output is down, because storage.pause_on_chunks_overlimit defaults to off for inputs. Each output's backlog is capped by storage.total_limit_size, after which the oldest chunks for that output are discarded. Retries wait a random time between scheduler.base (5 seconds) and scheduler.cap (2000 seconds), doubling the upper bound each attempt, per the scheduling docs.

Alert on the two counters that mean data loss. The chart renders these rules when the Prometheus Operator CRDs exist:

serviceMonitor:
  enabled: true
prometheusRule:
  enabled: true
  rules:
    - alert: FluentBitOutputRetriesFailing
      expr: sum by (pod, name) (rate(fluentbit_output_retries_failed_total[5m])) > 0
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Fluent Bit output {{ $labels.name }} on {{ $labels.pod }} is dropping chunks after retries"
    - alert: FluentBitRecordsDropped
      expr: sum by (pod, name) (rate(fluentbit_output_dropped_records_total[5m])) > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Fluent Bit output {{ $labels.name }} on {{ $labels.pod }} is dropping records"

Troubleshooting

Symptom	Likely cause	Fix
Pods Running, nothing arrives anywhere	--config still points at the classic fluent-bit.conf, whose default output is Elasticsearch at elasticsearch-master	Check args in the DaemonSet; it must end in fluent-bit.yaml
[warn] [input] tail.0 paused (mem buf overlimit ...)	Memory buffering and mem_buf_limit reached	Use storage.type: filesystem on the tail input
No kubernetes key; log says could not get meta for POD	RBAC or API server unreachable	kubectl auth can-i list pods --as=system:serviceaccount:logging:fluent-bit
Most pods enriched, a few never	Pod spec larger than buffer_size (32k default)	buffer_size: 0 on the kubernetes filter
Logs from short-lived Jobs missing	File deleted before the 60s default refresh_interval found it	refresh_interval: 10 or lower
Duplicate or missing lines after a restart	Tail db not on a hostPath, so offsets are lost	Put db under /var/fluent-bit/state
Stack traces arrive one line per record	Multiline filter missing, or placed after the kubernetes filter	Make multiline the first filter, multiline.key_content: log
OpenSearch: Action/metadata line [1] contains an unknown parameter [_type]	Type name sent to OpenSearch 2.0+	suppress_type_name: on
OpenSearch mapper_parsing_exception on kubernetes.labels.app	Dotted label keys clash with a plain app label	replace_dots: on
S3 AccessDenied	No IAM role reached the pod	Pod Identity association plus a rollout restart, or the IRSA annotation

When logs are missing for one pod, debug in the order the pipeline runs. First confirm the file exists on the node: kubectl debug node/$NODE -it --image=busybox:1.38.0 -- ls /host/var/log/containers | grep <pod>. The production image is distroless, so for a shell inside Fluent Bit itself, temporarily set image.tag: 5.1.3-debug. Then grep the Fluent Bit logs on that node for paused, could not get meta and error. Then compare fluentbit_input_records_total with fluentbit_output_proc_records_total and fluentbit_output_errors_total per output; a gap tells you which side is losing records. A pod that goes NotReady while a backend is down is expected: with health_check: on, /api/v2/health turns unhealthy once output errors or failed retries pass their thresholds within hc_period.

FAQ

Should I use Fluent Bit, Fluentd, Grafana Alloy or the OpenTelemetry Collector?

Fluent Bit is a small C binary built to run on every node with many native outputs, which is why it fits the DaemonSet role. Alloy makes sense if you are all-in on the Grafana stack, especially since Promtail reached end of life on 2 March 2026. The OpenTelemetry Collector is the choice when the rest of your pipeline is OTLP; see my Collector setup guide.

Is the classic Fluent Bit .conf format deprecated?

The Fluent Bit docs say classic files will be deprecated at the end of 2026, and YAML has been the standard format since v3.2. The Helm chart still generates a classic file by default, so override args and use config.extraFiles as shown in Step 2.

How do I exclude a namespace from Fluent Bit?

Enable namespace_exclude: on in the kubernetes filter and annotate the Namespace with fluentbit.io/exclude: "true"; a pod can opt back in with "false". For a hard exclusion that never reads the files, add a glob such as /var/log/containers/*_kube-system_*.log to exclude_path on the tail input.

Should I upgrade from Fluent Bit 4.2 to 5.1?

Yes, on a schedule. The 4.2 line's last release was 4.2.8 in August 2026, and 5.x is where fixes land. For a DaemonSet like this one the upgrade notes are short: update dashboards that use the hot-reload metric, and check any forward outputs to Fluentd.

In production the first thing I check is not the backend but the two counters that mean loss, failed retries and dropped records, followed by whether every node actually runs a Fluent Bit pod; a taint added for a new GPU pool is the classic way to lose a node's logs without a single error. The official references for everything above: the Kubernetes install page, the tail input, the kubernetes filter, buffering and storage, monitoring and the chart repository.

Related: How to Set Up Grafana Loki on Kubernetes with Alloy (2026), How to Set Up the OpenTelemetry Collector on Kubernetes (2026) and How to Fix Prometheus High Cardinality: Find and Drop Series (2026).