How to Run Chaos Experiments on Kubernetes with Chaos Mesh (2026)

This guide is for platform and SRE engineers who want to run real chaos experiments on Kubernetes without breaking anything they did not mean to break. By the end you will have Chaos Mesh installed and restricted to one opted-in namespace, three working experiments (pod kill, network latency, CPU and memory stress), a Prometheus-backed steady-state hypothesis that automatically aborts an experiment when the SLO is breached, a weekly scheduled experiment, and the same test expressed as a LitmusChaos ChaosEngine with a native Prometheus probe. Versions used: Chaos Mesh v2.8.5 (released 16 October 2026), the litmus-core Helm chart 3.31.1 with Litmus faults from chaos-charts 3.31.0, podinfo 6.15.0 as the target app, kube-prometheus-stack 92.10.1, and Helm 3.14. You need a non-production Kubernetes cluster that uses containerd, kubectl with cluster-admin for the install, and Helm 3. Every custom resource below was validated against the CRD schemas shipped in those exact chart versions.

If you want the background first (why chaos engineering, which tools, what the evidence says), read my companion post on chaos engineering in 2026.

Step 1: Install a target app and Prometheus

You need something to break and something to measure it with. podinfo is a small Go service that exposes http_requests_total{status} and http_request_duration_seconds histograms, which is all a steady-state hypothesis needs.

# Prometheus Operator stack. The selector flag makes Prometheus pick up
# ServiceMonitors from every namespace, not only ones labelled with the release.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add podinfo https://stefanprodan.github.io/podinfo
helm repo update

helm install kps prometheus-community/kube-prometheus-stack \
  --version 92.10.1 \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

# Target app: three replicas so killing one is survivable, with a ServiceMonitor.
kubectl create namespace shop
helm install podinfo podinfo/podinfo \
  --version 6.15.0 \
  --namespace shop \
  --set replicaCount=3 \
  --set serviceMonitor.enabled=true

Then add steady traffic, because an SLI with no requests is NaN, and a chaos experiment with no traffic proves nothing.

# loadgen.yaml: ten requests a second against podinfo
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loadgen
  namespace: shop
spec:
  replicas: 1
  selector:
    matchLabels: { app: loadgen }
  template:
    metadata:
      labels: { app: loadgen }
    spec:
      containers:
        - name: curl
          image: curlimages/curl:8.23.0
          command: ["/bin/sh", "-c"]
          args:
            - while true; do curl -s -o /dev/null -m 2 http://podinfo.shop:9898/; sleep 0.1; done

kubectl apply -f loadgen.yaml
kubectl -n shop get pods -l app.kubernetes.io/name=podinfo
# NAME                       READY   STATUS    RESTARTS   AGE
# podinfo-...                1/1     Running   0          1m   (three of these)

Step 2: Install Chaos Mesh 2.8.5 with Helm on containerd

The chart defaults chaosDaemon.runtime to docker and the socket to /var/run/docker.sock. Almost every cluster in 2026 runs containerd, so set both explicitly or network and stress experiments will fail to find the target container. The second important flag is controllerManager.enableFilterNamespace=true: with it, Chaos Mesh only injects into namespaces annotated chaos-mesh.org/inject=enabled. That is your first and cheapest blast-radius control.

helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update

# Socket paths from the Chaos Mesh 2.8.5 docs:
#   containerd (EKS, GKE, AKS, kubeadm, kind): /run/containerd/containerd.sock
#   k3s:       /run/k3s/containerd/containerd.sock
#   MicroK8s:  /var/snap/microk8s/common/run/containerd.sock
#   CRI-O:     runtime=crio, /var/run/crio/crio.sock
helm install chaos-mesh chaos-mesh/chaos-mesh \
  --version 2.8.5 \
  --namespace chaos-mesh --create-namespace \
  --set chaosDaemon.runtime=containerd \
  --set chaosDaemon.socketPath=/run/containerd/containerd.sock \
  --set controllerManager.enableFilterNamespace=true

kubectl get pods -n chaos-mesh -l app.kubernetes.io/instance=chaos-mesh
# Expect chaos-controller-manager, chaos-dashboard and one chaos-daemon
# per node, all Running.

Two defaults worth knowing. chaosDaemon.privileged is true, because the daemon enters container namespaces to run tc and stress-ng; that is why you install this in a dedicated namespace and never hand out chaos-mesh namespace access casually. And dashboard.securityMode defaults to true, which makes the dashboard ask for a Kubernetes token instead of acting with its own service account. Leave it on.

Now opt the target namespace in:

kubectl annotate namespace shop chaos-mesh.org/inject=enabled

Step 3: Define the steady-state hypothesis in Prometheus

Write the hypothesis before the fault. For podinfo, two SLIs cover most of what users feel:

# Error ratio over the last minute (target: at most 0.01)
(sum(rate(http_requests_total{namespace="shop",status=~"5.."}[1m])) or vector(0))
  /
sum(rate(http_requests_total{namespace="shop"}[1m]))

# p99 latency in seconds (target: at most 0.4)
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{namespace="shop"}[1m])))

Check both in the Prometheus UI (kubectl -n monitoring port-forward svc/prometheus-operated 9090) and confirm they sit well inside the target with the load generator running. One caveat: these are server-side metrics. A request that never reaches podinfo (connection refused during a pod kill) does not appear in http_requests_total. For production experiments, measure the SLI where the caller sees it: the ingress or gateway, or a client-side metric.

Chaos Mesh's StatusCheck only speaks HTTP and judges by status code, so wrap the PromQL in a tiny gate that returns 200 while the SLI is within budget and 503 when it is not. This is about 30 lines of standard-library Python; I tested it against a stub Prometheus returning 0.002 (200), 0.05 (503), NaN (503) and an empty result (503).

# slo-gate.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: slo-gate
  namespace: shop
data:
  gate.py: |
    # Returns 200 while the SLI is within budget, 503 otherwise.
    import json, os, urllib.parse, urllib.request
    from http.server import BaseHTTPRequestHandler, HTTPServer

    PROM = os.environ["PROM_URL"]
    QUERY = os.environ["SLI_QUERY"]
    MAX = float(os.environ["SLI_MAX"])

    def sli():
        url = f"{PROM}/api/v1/query?" + urllib.parse.urlencode({"query": QUERY})
        with urllib.request.urlopen(url, timeout=2) as r:
            result = json.load(r)["data"]["result"]
        return float(result[0]["value"][1]) if result else float("nan")

    class Handler(BaseHTTPRequestHandler):
        def do_GET(self):
            try:
                v = sli()
                ok = v == v and v <= MAX  # NaN (no traffic) counts as failure
            except Exception as e:
                v, ok = repr(e), False
            self.send_response(200 if ok else 503)
            self.end_headers()
            self.wfile.write(f"sli={v} max={MAX}\n".encode())

        def log_message(self, *args):
            pass

    HTTPServer(("", 8080), Handler).serve_forever()
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: slo-gate
  namespace: shop
  labels: { app: slo-gate }
spec:
  replicas: 1
  selector:
    matchLabels: { app: slo-gate }
  template:
    metadata:
      labels: { app: slo-gate }
    spec:
      containers:
        - name: gate
          image: python:3.13.16-slim
          command: ["python", "-I", "/app/gate.py"]
          env:
            - name: PROM_URL
              value: http://prometheus-operated.monitoring.svc:9090
            - name: SLI_MAX
              value: "0.01"
            - name: SLI_QUERY
              value: >-
                (sum(rate(http_requests_total{namespace="shop",status=~"5.."}[1m])) or vector(0))
                / sum(rate(http_requests_total{namespace="shop"}[1m]))
          ports:
            - containerPort: 8080
          volumeMounts:
            - { name: app, mountPath: /app }
      volumes:
        - name: app
          configMap: { name: slo-gate }
---
apiVersion: v1
kind: Service
metadata:
  name: slo-gate
  namespace: shop
spec:
  selector: { app: slo-gate }
  ports:
    - port: 8080
      targetPort: 8080

The gate pod lives in shop, which is opted in to chaos, but none of the selectors below match its app: slo-gate label. In a real setup put it in a namespace that is not annotated.

kubectl apply -f slo-gate.yaml
kubectl -n shop run gate-check --rm -i --restart=Never --image=curlimages/curl:8.23.0 -- \
  curl -s -w ' HTTP %{http_code}\n' http://slo-gate.shop.svc:8080/healthz
# sli=0.0 max=0.01 HTTP 200

Step 4: Kill a pod with PodChaos

Start with the smallest fault. mode: one picks a single random pod from the selector; pod-kill deletes it and lets the Deployment replace it.

# podchaos.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: podinfo-pod-kill
  namespace: shop
spec:
  action: pod-kill          # also: pod-failure (needs duration), container-kill
  mode: one                 # one | all | fixed | fixed-percent | random-max-percent
  selector:
    namespaces: ["shop"]
    labelSelectors:
      app.kubernetes.io/name: podinfo
  gracePeriod: 0            # seconds; 0 means immediate deletion

kubectl apply -f podchaos.yaml
kubectl -n shop get pods -l app.kubernetes.io/name=podinfo -w   # watch one pod restart
kubectl -n shop describe podchaos podinfo-pod-kill             # Events and Status.Experiment records
kubectl -n shop delete podchaos podinfo-pod-kill

Check the SLI queries while it runs. If the error ratio spiked, you have found your first finding: probably a missing readiness probe, a PodDisruptionBudget gap, or a client without retries.

Step 5: Add network latency with NetworkChaos

Latency finds more bugs than pod kills, because timeouts and retries are where distributed systems lie. This delays traffic leaving half the podinfo pods by 200 ms with 50 ms jitter, for five minutes.

# networkchaos.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: podinfo-latency
  namespace: shop
spec:
  action: delay             # also: loss, duplicate, corrupt, bandwidth, partition
  mode: fixed-percent
  value: "50"               # 50 percent of matching pods
  selector:
    namespaces: ["shop"]
    labelSelectors:
      app.kubernetes.io/name: podinfo
  direction: to
  delay:
    latency: "200ms"
    jitter: "50ms"
    correlation: "25"
  duration: "5m"

kubectl apply -f networkchaos.yaml
# Pause without deleting (keeps the object and its history):
kubectl -n shop annotate networkchaos podinfo-latency experiment.chaos-mesh.org/pause=true
# Resume:
kubectl -n shop annotate networkchaos podinfo-latency experiment.chaos-mesh.org/pause-

To scope latency to traffic towards one dependency, add a target selector (another pod selector plus mode) or externalTargets with hostnames or IPs. Without a target, every packet in the chosen direction is delayed, including DNS, which is often a finding in itself.

Step 6: Burn CPU and memory with StressChaos

StressChaos runs stress-ng inside the target container's cgroup, so it competes with your app for its own limits. Use it to check that CPU throttling and memory pressure degrade gracefully rather than cascade, and that your requests and limits are honest.

# stresschaos.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: podinfo-cpu-burn
  namespace: shop
spec:
  mode: one
  selector:
    namespaces: ["shop"]
    labelSelectors:
      app.kubernetes.io/name: podinfo
  stressors:
    cpu:
      workers: 1
      load: 80              # percent per worker
    memory:
      workers: 1
      size: "128MB"         # can also be a percentage of available memory
  duration: "3m"

Watch for the p99 SLI rising and for OOMKilled if the memory stressor pushes the pod over its limit. If that happens, the chaos tool did its job: you now know your limit is too tight for a noisy neighbour inside the same container.

Step 7: Gate the experiment on the SLO with a Workflow

Running a fault and watching dashboards is fine for a first run. For anything repeatable, let the cluster abort for you. A Chaos Mesh Workflow runs a StatusCheck in parallel with the fault; with abortWithStatusCheck: true, three consecutive failed checks abort the whole workflow and the fault is removed.

# workflow-gated.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: podinfo-latency-gated
  namespace: shop
spec:
  entry: entry
  templates:
    - name: entry
      templateType: Parallel
      deadline: 6m
      children:
        - slo-guard
        - inject-latency
    - name: slo-guard
      templateType: StatusCheck
      deadline: 6m
      abortWithStatusCheck: true      # failed gate aborts the workflow
      statusCheck:
        mode: Continuous
        type: HTTP
        intervalSeconds: 10
        timeoutSeconds: 3
        failureThreshold: 3
        successThreshold: 1
        http:
          url: http://slo-gate.shop.svc:8080/healthz
          method: GET
          criteria:
            statusCode: "200"
    - name: inject-latency
      templateType: NetworkChaos
      deadline: 5m
      networkChaos:
        action: delay
        mode: fixed-percent
        value: "50"
        selector:
          namespaces: ["shop"]
          labelSelectors:
            app.kubernetes.io/name: podinfo
        direction: to
        delay:
          latency: "200ms"
          jitter: "50ms"

kubectl apply -f workflow-gated.yaml
kubectl -n shop get workflow podinfo-latency-gated
kubectl -n shop get workflownode       # one node per template; check conditions
kubectl -n shop describe workflow podinfo-latency-gated

To prove the gate works, temporarily set SLI_MAX to 0 on the gate Deployment and rerun: the StatusCheck should fail three times and the workflow should abort. A guardrail you have never seen fire is a guardrail you do not have.

Step 8: Schedule a recurring experiment and limit who can run chaos

Once an experiment passes reliably, schedule it so regressions show up on a Tuesday afternoon rather than during an incident. concurrencyPolicy: Forbid stops a slow run from overlapping the next one.

# schedule.yaml: pod kill every Tuesday at 14:00 (cron is evaluated by the controller)
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: podinfo-weekly-pod-kill
  namespace: shop
spec:
  schedule: "0 14 * * 2"      # standard cron, or e.g. "@every 30m"
  type: PodChaos
  historyLimit: 5
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  podChaos:
    action: pod-kill
    mode: one
    selector:
      namespaces: ["shop"]
      labelSelectors:
        app.kubernetes.io/name: podinfo

Blast radius is also about people and agents. Give application teams a namespaced Role for Chaos Mesh resources in their own namespace only, and nothing in chaos-mesh:

# chaos-runner-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: chaos-runner
  namespace: shop
rules:
  - apiGroups: ["chaos-mesh.org"]
    resources: ["*"]
    verbs: ["get", "list", "watch", "create", "delete", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "list", "watch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: chaos-runner
  namespace: shop
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: chaos-runner
  namespace: shop
subjects:
  - kind: ServiceAccount
    name: chaos-runner
    namespace: shop
roleRef:
  kind: Role
  name: chaos-runner
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f chaos-runner-rbac.yaml
kubectl -n shop create token chaos-runner   # paste into the dashboard login

The same ServiceAccount is what I would hand an AI agent or a CI job that runs chaos, together with a policy that only allows pre-approved experiment templates. Kyverno or ValidatingAdmissionPolicy can enforce rules such as "no mode: all" and "duration at most 10m" on chaos-mesh.org resources.

Step 9: Run the same test with LitmusChaos and a Prometheus probe

LitmusChaos has one feature Chaos Mesh lacks: a native promProbe that evaluates PromQL during the experiment, so you do not need a gate service. You do not need the full ChaosCenter portal for this; the litmus-core chart installs only the chaos operator and its three CRDs (ChaosEngine, ChaosExperiment, ChaosResult).

helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm repo update
helm install litmus-core litmuschaos/litmus-core \
  --version 3.31.1 --namespace litmus --create-namespace

# The pod-delete fault definition (ChaosExperiment), pinned to 3.31.0:
kubectl -n shop apply -f \
  https://raw.githubusercontent.com/litmuschaos/chaos-charts/3.31.0/faults/kubernetes/pod-delete/fault.yaml

Create a ServiceAccount named pod-delete-sa in shop with the permissions listed in that fault.yaml (pods, pods/log, pods/exec, events, configmaps, jobs, apps resources and the litmuschaos.io resources), then apply the engine. PODS_AFFECTED_PERC: "34" hits one of three replicas, and stopOnFailure: true ends the run when the probe fails.

# litmus-engine.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: podinfo-pod-delete
  namespace: shop
spec:
  engineState: "active"
  appinfo:
    appns: "shop"
    applabel: "app.kubernetes.io/name=podinfo"
    appkind: "deployment"
  chaosServiceAccount: pod-delete-sa
  jobCleanUpPolicy: "delete"
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CHAOS_INTERVAL
              value: "15"
            - name: FORCE
              value: "false"
            - name: PODS_AFFECTED_PERC
              value: "34"
        probe:
          - name: error-ratio-under-1pct
            type: promProbe
            promProbe/inputs:
              endpoint: "http://prometheus-operated.monitoring.svc:9090"
              query: "(sum(rate(http_requests_total{namespace=\"shop\",status=~\"5..\"}[1m])) or vector(0)) / sum(rate(http_requests_total{namespace=\"shop\"}[1m]))"
              comparator:
                criteria: "<="
                value: "0.01"
            mode: Continuous            # SOT | EOT | Edge | Continuous | OnChaos
            runProperties:
              probeTimeout: 5s
              interval: 10s
              retry: 2
              probePollingInterval: 10s
              stopOnFailure: true

kubectl apply -f litmus-engine.yaml
kubectl -n shop get chaosresult podinfo-pod-delete-pod-delete -o jsonpath='{.status.experimentStatus.verdict}{"\n"}'
# Pass, Fail or Awaited

If you want the portal, the litmus chart (3.30.0 at the time of writing) installs ChaosCenter with a MongoDB dependency; the installation docs cover it, and you should change the default admin credentials immediately.

Troubleshooting

Symptom	Likely cause	Fix
NetworkChaos or StressChaos stuck, daemon logs mention docker.sock	Chart default runtime is docker	Reinstall or upgrade with chaosDaemon.runtime=containerd and the correct socketPath for your distro (Step 2)
Experiment created but no pods affected, records show nothing injected	enableFilterNamespace=true and target namespace not annotated	kubectl annotate ns shop chaos-mesh.org/inject=enabled
Admission webhook rejects the resource	Invalid enum or missing field, for example latency missing or mode misspelt	kubectl apply --dry-run=server -f file.yaml and fix the field it names
Webhook timeouts on GKE private clusters	Control plane cannot reach the webhook port	Chart already uses port 10250 for webhooks; check firewall rules if you changed WEBHOOK_PORT
Controller logs say it cannot reach chaos-daemon	NetworkPolicy blocking the chaos-mesh namespace	Allow controller-to-daemon traffic; the FAQ also suggests checking the pod network
Deleting an experiment hangs forever	Finalizer waiting for recovery on a pod that is gone	kubectl annotate <kind> <name> -n shop chaos-mesh.chaos-mesh.org/cleanFinalizer=forced, then delete
chaos-daemon blocked on OpenShift with SCC errors	Daemon needs hostPID, hostNetwork and privileged	oc adm policy add-scc-to-user privileged -n chaos-mesh -z chaos-daemon
Upgrade fails with unknown fields	Helm does not upgrade CRDs	Apply the CRDs for the new version manually before helm upgrade
Workflow StatusCheck never aborts	Wrong gate URL, abortWithStatusCheck missing, or a gate that does not fail closed on NaN	Curl the gate from inside the namespace, set abortWithStatusCheck: true, and test by setting SLI_MAX to 0
Litmus verdict Awaited forever	Operator not running or chaosServiceAccount lacks permissions	kubectl -n litmus get pods, then check runner pod logs in shop for RBAC errors

FAQ

Is Chaos Mesh safe to install on a production cluster?

The chaos-daemon runs privileged on every node, so treat the chaos-mesh namespace like any other cluster-admin component. With enableFilterNamespace=true, dashboard.securityMode on and namespaced RBAC for users, it is reasonable to install in production, but run each experiment in staging first.

How do I stop a Chaos Mesh experiment immediately?

Delete the resource with kubectl delete, which triggers recovery, or annotate it with experiment.chaos-mesh.org/pause=true to pause while keeping its history. If deletion hangs, add the chaos-mesh.chaos-mesh.org/cleanFinalizer=forced annotation and delete again.

Does Chaos Mesh integrate with Prometheus?

Chaos Mesh exposes its own controller metrics, but its StatusCheck only evaluates HTTP status codes, so to gate on PromQL you need a small HTTP wrapper like the one in Step 3. LitmusChaos has a native promProbe that runs PromQL directly.

Chaos Mesh or LitmusChaos: which should I use?

Use Chaos Mesh if you want plain CRDs, a small footprint and GitOps-friendly YAML. Use Litmus if you want Prometheus probes, a portal and a hub of reusable faults, and do not mind running MongoDB for ChaosCenter. Both are CNCF incubating projects with releases in October 2026.

When I set this up for a team, the first thing I check is not the fault but the gate: does the steady-state query return a sane number with traffic, does it fail closed on NaN, and have we watched it abort a run at least once. Then I opt in exactly one namespace and start with mode: one. The official references are the Chaos Mesh docs, the Chaos Mesh Helm installation guide, the LitmusChaos probe documentation and the Principles of Chaos.

Related: Chaos Engineering in 2026: Tools, AI Agents and What the Evidence Says, AIOps in 2026: 47% on ITBench, 10% on Hard On-Call, $6.50 an Investigation, and Four Outages Where the Automation Was the Incident and Observability in 2026: OpenTelemetry Graduated, the Collector Is Still v0.162, eBPF Instrumentation Is v0.14, and Datadog Bills $1.12B a Quarter.