How to Set Up Grafana Loki on Kubernetes with Alloy (2026)

This guide is for platform engineers who want a working, production-shaped Grafana Loki on Kubernetes: logs collected from every pod by Grafana Alloy, stored in S3, queryable from Grafana with LogQL, with retention that actually deletes data and limits that will not page you at 2 AM. At the end you will have a Monolithic Loki 3.7.8 installed from the grafana-community/loki Helm chart 18.17.1, Alloy v1.21.0 from the grafana/alloy chart 1.14.0 running as a DaemonSet, and a short list of fixes for the errors everyone hits. I rendered both charts with Helm 3.14.4 and helm template against these exact values on 20 October 2026; the cluster side follows the official docs. Prerequisites: a Kubernetes cluster you can install into, Helm 3, kubectl, an S3 bucket (or GCS/Azure equivalent), and a Grafana instance.

Two things changed this year that break older tutorials. Promtail reached end of life on 2 March 2026, so the collector is now Alloy. And the open-source Loki chart moved on 16 March 2026 to the grafana-community/helm-charts repository; the grafana/loki chart in the old repo is now for Grafana Enterprise Logs users. If you copy helm install loki grafana/loki from a 2025 blog post, you get the wrong chart.

Step 1: Choose a Loki deployment mode

The community chart supports three modes, set with deploymentMode. The chart's own values.yaml describes them like this:

Mode	Shape	Rough scale (per chart docs)	Status
Monolithic (formerly SingleBinary)	One StatefulSet running every component	Up to a few tens of GB/day	Default, recommended for small stacks
SimpleScalable	read, write, backend targets	Up to about 1 TB/day	Deprecated, removed in Loki 4
Distributed	Each microservice as its own workload	Over 1 TB/day	What Grafana recommends for production at scale

The Loki Helm docs say plainly that Simple Scalable "is being deprecated" and point production users at microservices (Distributed) mode. My advice: start Monolithic unless you already know you ingest hundreds of GB a day. The storage schema is identical across modes and the chart has migration modes (Monolithic<->SimpleScalable, SimpleScalable<->Distributed), so you are not locked in. Do not start a new install on Simple Scalable in 2026.

Step 2: Create object storage and an IAM role

Loki stores chunks and its TSDB index in object storage; local disk is only for the write-ahead log and caches. Create three buckets with unique names. The install docs warn against the generic names chunks, ruler and admin on S3, because bucket names are global.

# AWS CLI v2. Pick your own globally unique prefix.
export AWS_REGION=eu-west-1
for b in acme-loki-chunks acme-loki-ruler acme-loki-admin; do
  aws s3api create-bucket --bucket "$b" --region "$AWS_REGION" \
    --create-bucket-configuration LocationConstraint="$AWS_REGION"
  aws s3api put-public-access-block --bucket "$b" \
    --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
done

Give Loki an IAM role through IRSA or EKS Pod Identity instead of static keys. The policy needs list, get, put and delete on the buckets:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
      "Resource": [
        "arn:aws:s3:::acme-loki-chunks", "arn:aws:s3:::acme-loki-chunks/*",
        "arn:aws:s3:::acme-loki-ruler",  "arn:aws:s3:::acme-loki-ruler/*",
        "arn:aws:s3:::acme-loki-admin",  "arn:aws:s3:::acme-loki-admin/*"
      ]
    }
  ]
}

s3:DeleteObject is not optional: without it the compactor cannot enforce retention and your bucket grows forever. If you federate identities across clouds, the same pattern is covered in workload identity federation explained.

Avoid the bundled MinIO subchart. The chart marks it deprecated, and the monolithic install page notes it is scheduled for removal on 31 October 2026.

Step 3: Install Loki with the grafana-community Helm chart

Save this as loki-values.yaml. Every key below exists in chart 18.17.1; the comments explain the ones that matter.

# grafana-community/loki 18.17.1 (appVersion 3.7.8)
deploymentMode: Monolithic

loki:
  auth_enabled: false            # single tenant; true requires an X-Scope-OrgID header on every request
  commonConfig:
    replication_factor: 1        # 3 if you run 3 replicas
  schemaConfig:
    configs:
      - from: "2024-04-01"       # must be a date in the past for a new install
        store: tsdb
        object_store: s3
        schema: v13              # v13 + tsdb is required for structured metadata
        index:
          prefix: loki_index_
          period: 24h            # retention only works with a 24h index period
  storage:
    type: s3
    bucketNames:
      chunks: acme-loki-chunks
      ruler: acme-loki-ruler
      admin: acme-loki-admin
    s3:
      region: eu-west-1          # credentials come from the IRSA role below
  limits_config:
    retention_period: 744h       # 31 days, enforced by the compactor
    allow_structured_metadata: true
    volume_enabled: true
    ingestion_rate_mb: 8         # default 4
    ingestion_burst_size_mb: 16  # default 6
    max_global_streams_per_user: 10000   # default 5000
    split_queries_by_interval: 1h        # chart sets 15m; see Troubleshooting
  compactor:
    retention_enabled: true
    delete_request_store: s3     # required when retention is enabled
    retention_delete_delay: 2h
  pattern_ingester:
    enabled: true                # powers pattern detection in Grafana Logs Drilldown

singleBinary:
  replicas: 1
  persistence:
    size: 30Gi
  resources:
    requests: { cpu: "1", memory: 2Gi }
    limits:   { memory: 4Gi }

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/loki-s3

chunksCache:
  allocatedMemory: 1024          # default 8192 MB asks for ~9.8 GiB of RAM
resultsCache:
  allocatedMemory: 512

# Scale the other modes' components to zero in Monolithic mode
write:   { replicas: 0 }
read:    { replicas: 0 }
backend: { replicas: 0 }

Install it:

helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
helm upgrade --install loki grafana-community/loki \
  --version 18.17.1 --namespace loki --create-namespace \
  -f loki-values.yaml

The rendered release contains a loki StatefulSet, loki-chunks-cache and loki-results-cache memcached StatefulSets, a loki-gateway Deployment (nginx, Service on port 80), and a loki-canary DaemonSet that writes and reads synthetic lines so you can alert on Loki itself.

If you are upgrading from the old grafana/loki 6.x chart, read the upgrade to community chart page first. Version 6.55.0 was the last open-source release from the Loki repo, and the community chart went through several major versions with breaking changes, including the SingleBinary to Monolithic rename in 12.0.0.

Step 4: Verify Loki is ingesting and answering queries

kubectl -n loki get pods
# Expect loki-0 2/2 Running, loki-gateway-..., loki-chunks-cache-0, loki-results-cache-0, loki-canary-...

kubectl -n loki port-forward svc/loki-gateway 3100:80 &

# Push one line. Timestamps are nanoseconds; on macOS use: $(($(date +%s)*1000000000))
curl -s -X POST http://localhost:3100/loki/api/v1/push \
  -H "Content-Type: application/json" \
  --data-raw "{\"streams\":[{\"stream\":{\"job\":\"smoke\"},\"values\":[[\"$(date +%s%N)\",\"hello loki\"]]}]}"

# Read it back
curl -s -G http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="smoke"}' | jq '.data.result[0].values'

A successful push returns HTTP 204 with an empty body. In Grafana, add a Loki data source with URL http://loki-gateway.loki.svc.cluster.local. If you set auth_enabled: true, add an X-Scope-OrgID header to the data source.

Step 5: Collect pod logs with Grafana Alloy

Alloy replaces Promtail, the Grafana Agent and several other collectors with one binary. Grafana announced the Promtail merge into Alloy with Loki 3.4 in February 2025; LTS ended with EOL on 2 March 2026. If you have an existing Promtail config, convert it rather than rewriting it:

alloy convert --source-format=promtail --output=config.alloy promtail.yaml
# Add --report=report.txt to see what did not convert cleanly

For a fresh install, this alloy-values.yaml runs Alloy as a DaemonSet and tails pod logs through the Kubernetes API with loki.source.kubernetes, which needs no hostPath mounts and no root. Each Alloy pod only discovers pods on its own node; the chart sets HOSTNAME to spec.nodeName, which the field selector uses.

# grafana/alloy chart 1.14.0 (Alloy v1.21.0)
controller:
  type: daemonset
alloy:
  configMap:
    content: |
      // Discover pods scheduled on this node only
      discovery.kubernetes "pods" {
        role = "pod"
        selectors {
          role  = "pod"
          field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
        }
      }

      // Keep labels few and bounded: namespace, container, app
      discovery.relabel "pods" {
        targets = discovery.kubernetes.pods.targets
        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          target_label  = "app"
        }
      }

      loki.source.kubernetes "pods" {
        targets    = discovery.relabel.pods.output
        forward_to = [loki.process.pods.receiver]
      }

      loki.process "pods" {
        stage.static_labels {
          values = { cluster = "prod-eu-1" }
        }
        // Lift trace_id out of JSON lines into structured metadata, not a label
        stage.json {
          expressions = { trace_id = "trace_id" }
        }
        stage.structured_metadata {
          values = { trace_id = "" }
        }
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki-gateway.loki.svc.cluster.local/loki/api/v1/push"
        }
      }

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install alloy grafana/alloy \
  --version 1.14.0 --namespace alloy --create-namespace -f alloy-values.yaml
kubectl -n alloy port-forward ds/alloy 12345:12345   # Alloy UI: component health graph

The chart's ClusterRole already grants pods and pods/log. The trade-off with loki.source.kubernetes is extra load on the kubelet and API path; on very busy nodes, switch to loki.source.file reading /var/log/pods with alloy.mounts.varlog: true. The same Alloy DaemonSet can also collect profiles, which I touch on in continuous profiling in 2026.

Step 6: Labels vs structured metadata

This is the decision that determines whether Loki stays cheap. Labels define streams, and every unique label combination is a separate stream with its own chunks and index entries. max_global_streams_per_user defaults to 5,000, which a single high-cardinality label (pod name, user ID, request ID) can blow through in minutes.

Rules I apply:

• Labels for things you always filter on first and that have bounded values: cluster, namespace, container, app, maybe env. Loki also derives a service_name label automatically from app, container and similar labels (the discover_service_name default list).
• Structured metadata for high-cardinality values you sometimes filter on: trace_id, pod, user_id. It is stored next to each line, not in the index. It requires schema v13 with TSDB and allow_structured_metadata: true, which is the default in 3.x, and it is also how native OTLP ingestion stores resource attributes (docs). Limits: 64 KB and 128 entries per line.
• Leave it in the line for everything else, and parse at query time with | json or | logfmt.

If you would not put it in a Prometheus label, do not put it in a Loki label. The same cardinality reasoning is in the Prometheus high cardinality guide.

Step 7: LogQL basics you will use daily

# 1. Stream selector + line filter (fastest: filters before parsing)
{namespace="shop", container="api"} |= "timeout" != "healthz"

# 2. Parse JSON and filter on a field
{namespace="shop", container="api"} | json | status >= 500

# 3. Filter on structured metadata, no parsing needed
{namespace="shop"} | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"

# 4. Error rate per container over 5 minutes (metric query)
sum by (container) (rate({namespace="shop"} |= "error" [5m]))

# 5. Noisiest workloads by line volume in the last hour
topk(10, sum by (namespace, container) (count_over_time({cluster="prod-eu-1"}[1h])))

# 6. Reformat logfmt output for reading
{app="payments"} | logfmt | line_format "{{.level}} {{.msg}}"

Two habits keep queries fast: always start with the narrowest stream selector you can, and put line filters (|=, !=, |~) before parsers. Metric queries return series, and the default max_query_series is 500, so sum by (pod) over a busy cluster will fail; aggregate on a coarser label.

Step 8: Configure retention and per-tenant limits

Retention is done by the compactor, not by S3 lifecycle rules. The retention docs require compactor.retention_enabled: true, a delete_request_store, and a 24h index period, all of which the values in Step 3 set. Chunks are marked, then deleted after retention_delete_delay (default 2h). Per-stream retention lets you keep noisy debug logs for less time:

loki:
  limits_config:
    retention_period: 744h
    retention_stream:
      - selector: '{namespace="dev"}'
        priority: 1
        period: 72h          # each retention_stream period must be at least 24h
      - selector: '{container="audit"}'
        priority: 2
        period: 2160h        # 90 days

Do not also set an S3 lifecycle expiry shorter than retention_period; deleting chunks underneath the index produces query errors for missing objects. The legacy Table Manager retention is deprecated.

Troubleshooting

The ingestion errors below map to limits_config keys documented on the rate limits and validation page; defaults are from the Loki 3.7.8 configuration reference.

Symptom	Likely cause	Fix
Grafana panel: too many outstanding requests (HTTP 429) on long time ranges	Query split into more sub-queries than the queue allows. The chart sets split_queries_by_interval: 15m, so 30 days is 2,880 sub-queries against a frontend default max_outstanding_per_tenant of 2048	Set split_queries_by_interval: 1h; raise loki.frontend.max_outstanding_per_tenant (and loki.query_scheduler.max_outstanding_requests_per_tenant, default 32000, if you run a scheduler); add querier capacity
Alloy logs 429 with reason rate_limited	Tenant over ingestion_rate_mb (default 4) or ingestion_burst_size_mb (default 6)	Raise both, then find the noisy app with query 5 in Step 7
429 reason per_stream_rate_limit	One stream over 3 MB/s (burst 15 MB)	Add a bounded label to split the stream, or raise per_stream_rate_limit
429 reason stream_limit	More than max_global_streams_per_user (5,000) active streams	Remove high-cardinality labels; move them to structured metadata
400 greater_than_max_sample_age after Alloy restart	Replaying old lines older than reject_old_samples_max_age (chart sets 168h)	Expected for very old logs; otherwise raise the max age
400 too_far_behind	Out-of-order lines older than the allowed window for a stream	Usually duplicate collectors on one node; check the node field selector
maximum of series (500) reached for a single query	Metric query grouped by a high-cardinality label	Aggregate by a coarser label, or raise max_query_series carefully
no org id (HTTP 401)	auth_enabled: true without an X-Scope-OrgID header	Add the header in Grafana and set tenant_id in loki.write, or disable auth
loki-chunks-cache-0 stuck Pending	Default chunksCache.allocatedMemory: 8192 requests about 9.8 GiB	Lower allocatedMemory or add a larger node
Logs arrive but old data never deletes	Compactor retention off, or IAM missing s3:DeleteObject	Set retention_enabled: true and delete_request_store; fix the policy

For pods that will not schedule at all, see Kubernetes pod Pending troubleshooting.

FAQ

Is Promtail still supported in 2026?

No. Promtail reached end of life on 2 March 2026 and receives no further updates. Migrate to Grafana Alloy; alloy convert --source-format=promtail translates most configurations, though metrics names, positions file location and some CLI flags differ.

Which Loki Helm chart should I use now?

For open-source Loki, use grafana-community/loki from https://grafana-community.github.io/helm-charts. The grafana/loki chart in Grafana's repo is maintained for Grafana Enterprise Logs. Chart 18.17.1 ships Loki 3.7.8.

Should I use Simple Scalable or Distributed mode?

For new installs, neither unless you need the scale. Start Monolithic; move to Distributed when ingest grows past what one StatefulSet handles. Simple Scalable is deprecated and the chart says it will be removed in Loki 4.

What is the difference between labels and structured metadata in Loki?

Labels are indexed and define streams, so they must be low cardinality. Structured metadata is stored with each log line, is not indexed, and suits values like trace IDs and pod names that you filter on occasionally. Both can be used in LogQL; structured metadata filters go after the stream selector.

When a Loki install misbehaves, the first thing I check is the stream count and the labels behind it, because most Loki ingestion and cost problems trace back to one label someone added without thinking about cardinality. The second is whether the compactor is actually deleting anything. In production I run the canary, alert on its missing-entries metric, and keep the values file in Git so every limit change goes through review like any other infrastructure change. Official references: the Loki Helm installation docs, the Loki configuration reference, the Alloy collect-logs-in-Kubernetes guide and the Promtail migration guide.

Related: Observability in 2026: OpenTelemetry Graduated, the Collector Is Still v0.162, eBPF Instrumentation Is v0.14, and Datadog Bills $1.12B a Quarter, It's 2 AM. Do You Know What Your AI Agent Is Doing? and Agent Observability in 2026.