How to Set Up HolmesGPT and K8sGPT for Kubernetes Troubleshooting (2026)

This guide is for platform engineers and SREs who want to try AI troubleshooting on a real cluster without handing a language model their admin kubeconfig. By the end you will have K8sGPT 0.4.41 and HolmesGPT 0.45.0 installed at pinned versions, both running through a dedicated read-only ServiceAccount, connected to a hosted model or a local Ollama model, investigating a deliberately broken pod, and HolmesGPT pulling alerts from Alertmanager. I checked every command and flag against the projects' docs and source at those tags as of 31 October 2026. You need kubectl access to a test cluster (kind, minikube or a dev namespace on a real cluster is fine), permission to create a ClusterRole, and an API key for one LLM provider or a machine that can run Ollama.

If you want the comparison and the reasoning behind the safety choices first, read K8sGPT vs HolmesGPT vs kubectl-ai. This post is the hands-on half.

Step 1: Create a read-only ServiceAccount for AI tools

Both tools use whatever kubeconfig they are given. HolmesGPT's Kubernetes toolset docs say the CLI uses your current kubectl context, so if that context is cluster-admin, so is the agent. Fix that first, with a ClusterRole that can read what both tools' analyzers and toolsets need and nothing else.

# ai-readonly.yaml  (tested against Kubernetes 1.30+; plain RBAC v1)
apiVersion: v1
kind: Namespace
metadata:
  name: ai-troubleshoot
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ai-readonly
  namespace: ai-troubleshoot
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-troubleshooter-readonly
rules:
  # Core objects. No "secrets" on purpose. configmaps is needed by
  # K8sGPT's ConfigMap analyzer; drop it if yours hold credentials.
  - apiGroups: [""]
    resources: ["pods", "pods/log", "events", "services", "endpoints",
                "nodes", "namespaces", "persistentvolumeclaims",
                "persistentvolumes", "configmaps", "replicationcontrollers",
                "serviceaccounts", "resourcequotas", "limitranges"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses", "networkpolicies"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["autoscaling"]
    resources: ["horizontalpodautoscalers"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["admissionregistration.k8s.io"]
    resources: ["mutatingwebhookconfigurations", "validatingwebhookconfigurations"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["events.k8s.io", "discovery.k8s.io"]
    resources: ["events", "endpointslices"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ai-troubleshooter-readonly
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ai-troubleshooter-readonly
subjects:
  - kind: ServiceAccount
    name: ai-readonly
    namespace: ai-troubleshoot

Apply it with your normal admin context, then mint a short-lived token and build a kubeconfig that contains only that token:

kubectl apply -f ai-readonly.yaml

# Short-lived token (the API server may cap the maximum duration)
TOKEN=$(kubectl -n ai-troubleshoot create token ai-readonly --duration=8h)

# Copy the cluster endpoint and CA from your current context
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
kubectl config view --minify --raw \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 --decode > /tmp/ai-ca.crt

export AI_KUBECONFIG="$HOME/.kube/ai-readonly.yaml"
kubectl --kubeconfig="$AI_KUBECONFIG" config set-cluster target \
  --server="$SERVER" --certificate-authority=/tmp/ai-ca.crt --embed-certs=true
kubectl --kubeconfig="$AI_KUBECONFIG" config set-credentials ai-readonly --token="$TOKEN"
kubectl --kubeconfig="$AI_KUBECONFIG" config set-context ai-readonly --cluster=target --user=ai-readonly
kubectl --kubeconfig="$AI_KUBECONFIG" config use-context ai-readonly
chmod 600 "$AI_KUBECONFIG"

Prove it is read-only before you trust it:

kubectl --kubeconfig="$AI_KUBECONFIG" auth can-i list pods -A        # yes
kubectl --kubeconfig="$AI_KUBECONFIG" auth can-i delete pods -A      # no
kubectl --kubeconfig="$AI_KUBECONFIG" auth can-i get secrets -A      # no
kubectl --kubeconfig="$AI_KUBECONFIG" auth can-i patch deployments -A # no

If any of the last three says yes, stop and check for other bindings on that ServiceAccount. If you hit Forbidden errors later, my RBAC Forbidden fix guide walks through reading them.

Step 2: Install K8sGPT 0.4.41 (pinned)

Homebrew (brew install k8sgpt) always gives you the latest, which is fine on a laptop. For anything scripted, pin the release tarball and check it against the published checksums from the v0.4.41 release:

K8SGPT_VERSION=0.4.41
OS=Linux          # or Darwin
ARCH=x86_64       # or arm64
FILE="k8sgpt_${OS}_${ARCH}.tar.gz"

curl -fsSLO "https://github.com/k8sgpt-ai/k8sgpt/releases/download/v${K8SGPT_VERSION}/${FILE}"
curl -fsSLO "https://github.com/k8sgpt-ai/k8sgpt/releases/download/v${K8SGPT_VERSION}/checksums.txt"
grep " ${FILE}\$" checksums.txt | sha256sum -c -    # macOS: shasum -a 256 -c -
tar -xzf "$FILE" k8sgpt
install -m 0755 k8sgpt "$HOME/.local/bin/k8sgpt"   # any directory on your PATH
k8sgpt version

Run a scan with no AI backend at all. This is the cheapest, most private thing either tool does, and it is worth keeping even after you add a model:

k8sgpt analyze --kubeconfig="$AI_KUBECONFIG"
# Only pod problems in one namespace, as JSON
k8sgpt analyze --kubeconfig="$AI_KUBECONFIG" --filter=Pod --namespace=default --output=json

k8sgpt filters list shows which analyzers are active. Only the core set (Pod, Service, Deployment, PVC, Node, Ingress, webhooks, ConfigMap and a few more) runs by default; optional ones such as HPA, PodDisruptionBudget, NetworkPolicy, Gateway API and Log need k8sgpt filters add <name>, per the README.

Step 3: Connect K8sGPT to an LLM or a local model

K8sGPT only contacts a model when you pass --explain. Add a backend with k8sgpt auth add; it prompts for the key rather than taking it on the command line, which keeps it out of your shell history. The backend reference lists every provider.

# Hosted: OpenAI (prompts for the API key)
k8sgpt auth add --backend openai --model gpt-4o-mini

# Local: Ollama on its default port (pull a model first: ollama pull llama3.1)
k8sgpt auth add --backend ollama --model llama3.1 --baseurl http://localhost:11434
k8sgpt auth default -p ollama

k8sgpt auth list

Now ask for explanations, masked where possible:

k8sgpt analyze --kubeconfig="$AI_KUBECONFIG" --explain --anonymize --filter=Pod -n default

Two privacy facts from the README you should know before pointing this at production. Anonymization masks object names and labels for analyzers such as Deployment, Service, StatefulSet and Node, but it does not apply to events, and the Pod, ReplicaSet, PVC and Log analyzers are not masked. And the config file, including your API key, is plain text at ~/.config/k8sgpt/k8sgpt.yaml on Linux or ~/Library/Application Support/k8sgpt/k8sgpt.yaml on macOS.

Step 4: Install HolmesGPT 0.45.0 (pinned)

HolmesGPT 0.45.0 on PyPI declares Python >=3.10,<3.14. pipx keeps it out of your system Python:

# Pin both the package and a supported interpreter
pipx install --python python3.12 "holmesgpt==0.45.0"
holmes version
holmes ask --help | head -20

Homebrew also works (brew tap robusta-dev/homebrew-holmesgpt && brew install holmesgpt, per the install docs) but tracks the latest release.

Step 5: Configure HolmesGPT's model and guardrails

HolmesGPT reads ~/.holmes/config.yaml, with command-line flags taking precedence. Set the model, and tighten the bash toolset, which is enabled by default and lets the agent run shell commands after an allowlist check and your approval:

# ~/.holmes/config.yaml  (HolmesGPT 0.45.0)
model: "gpt-4.1"            # or "anthropic/claude-sonnet-4-5-20250929", "ollama_chat/<model>"
max_steps: 40               # default is 100; lower caps cost and wandering
toolsets:
  bash:
    enabled: true
    config:
      builtin_allowlist: "core"   # "none", "core" or "extended"
      deny:
        - "kubectl get secret"
        - "kubectl describe secret"
        - "kubectl delete"
        - "kubectl apply"

Then export the provider key and point the CLI at the read-only kubeconfig:

export OPENAI_API_KEY="..."            # or ANTHROPIC_API_KEY, GEMINI_API_KEY
export KUBECONFIG="$AI_KUBECONFIG"     # HolmesGPT shells out to kubectl with this

# Local model instead (experimental per the docs):
# export OLLAMA_API_BASE="http://localhost:11434"
# holmes ask "what pods are failing?" --model="ollama_chat/llama3.1"

The Ollama page is candid that tool calling with local models is limited and inconsistent, and recommends confirming everything works with a hosted model first. Take that advice; otherwise you cannot tell a setup problem from a model problem.

Step 6: Break a pod and investigate it with both tools

Create two classic failures in a scratch namespace (use your admin context for this, not the AI one):

kubectl create namespace ai-demo
kubectl -n ai-demo create deployment bad-image --image=nginx:1.27.999-does-not-exist
kubectl -n ai-demo apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: stuck-pending
spec:
  nodeSelector:
    disktype: unicorn        # no node has this label
  containers:
    - name: app
      image: nginx:1.27
      resources:
        requests: { cpu: 100m, memory: 64Mi }
EOF

K8sGPT first:

k8sgpt analyze --kubeconfig="$AI_KUBECONFIG" -n ai-demo --filter=Pod --explain

You should see one finding per broken pod: an image pull back-off for bad-image and a scheduling failure for stuck-pending mentioning the node selector, each followed by the model's explanation and suggested fix. The exact wording varies by model.

Now HolmesGPT, which will decide for itself to describe the pod, read events and check node labels:

holmes ask "why is the stuck-pending pod in namespace ai-demo not running?" --no-interactive
holmes ask "what is wrong with deployment bad-image in ai-demo, and what is the fix?" \
  --no-interactive --log-costs --show-tool-output

--show-tool-output prints every command it ran, which is the fastest way to learn whether it is investigating or guessing. --log-costs prints LLM spend. The diagnosis should name the missing disktype=unicorn label and the nonexistent image tag. Neither tool can fix them, because the identity they hold cannot write. That is the point. These two failure types have their own long-form guides on this site: Pending pods and ImagePullBackOff.

Step 7: Investigate Alertmanager alerts with HolmesGPT

holmes investigate alertmanager fetches firing alerts and investigates each one. Port-forward Alertmanager (the service is alertmanager-operated in a kube-prometheus-stack install; adjust for yours):

kubectl -n monitoring port-forward svc/alertmanager-operated 9093:9093

To test without waiting for a real alert, post one. Use the v2 API: Alertmanager 0.27.0 removed every /api/v1/ endpoint, and they now answer 410 Gone, which catches out older tutorials.

curl -fsS -X POST http://localhost:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[{
        "labels": {"alertname": "KubePodNotReady", "severity": "warning",
                   "namespace": "ai-demo", "pod": "stuck-pending"},
        "annotations": {"summary": "Pod ai-demo/stuck-pending has been Pending"},
        "startsAt": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"
      }]'

Then investigate only that alert, or filter by label:

holmes investigate alertmanager --alertmanager-url http://localhost:9093 \
  --alertmanager-alertname "KubePodNotReady"

# Only critical alerts in production, at most 3 per run
holmes investigate alertmanager --alertmanager-url http://localhost:9093 \
  --alertmanager-label "severity=critical" --alertmanager-label "namespace=production" -n 3

Repeated --alertmanager-label flags must all match. Add --destination slack --slack-token ... --slack-channel "#oncall" once you trust the output; the source marks Slack output as experimental. Start with one route. Running an LLM over every firing alert in a noisy cluster is an expensive way to rediscover alert fatigue, which this Alertmanager guide fixes more cheaply.

Step 8: Decide what data leaves the cluster

Write this down for your security team rather than leaving it implicit:

• K8sGPT without --explain: nothing leaves your machine.
• K8sGPT with --explain: analyzer findings go to the provider, including event messages and pod names even with --anonymize.
• HolmesGPT: every tool output the agent reads goes to the provider, which can include pod logs, events, metrics and ConfigMap contents. Its Kubernetes toolset excludes Secrets by default, and your RBAC role excludes them regardless.
• Either with Ollama or a self-hosted OpenAI-compatible endpoint: data stays where the model runs.

If you route hosted models through a gateway for budgets and logging, both tools can use one: K8sGPT has a litellm backend and HolmesGPT is built on LiteLLM model names. My LiteLLM proxy guide covers that setup.

Troubleshooting

Symptom	Likely cause	Fix
Forbidden: User "system:serviceaccount:ai-troubleshoot:ai-readonly" cannot list resource ...	Role is missing a resource or API group the analyzer or tool needs	Add that resource with get, list, watch to the ClusterRole; never add secrets or *
error: You must be logged in to the server (Unauthorized) after a few hours	The kubectl create token token expired	Re-run Step 1's token and set-credentials lines; automate it in a wrapper script
k8sgpt analyze --explain errors about a missing or default backend	No backend configured, or the default is not the one you added	k8sgpt auth list, then k8sgpt auth default -p <backend> or pass --backend
K8sGPT with Ollama returns connection refused	Ollama not running or listening on another host or port	ollama serve, check curl http://localhost:11434/api/tags, match --baseurl
pipx install holmesgpt==0.45.0 fails resolving	Default Python is 3.14 or older than 3.10	pipx install --python python3.12 "holmesgpt==0.45.0"
HolmesGPT says it cannot find the pod or namespace you can see	KUBECONFIG not exported in that shell, or points at another context	echo $KUBECONFIG, kubectl --kubeconfig="$AI_KUBECONFIG" get pods -A
HolmesGPT keeps prompting to approve bash commands	Command not in the bash allowlist	Approve once (saved to ~/.holmes/bash_approved_prefixes.yaml), add it to allow, or use --bash-always-deny for unattended runs
Ollama model loops or returns no tool calls in HolmesGPT	Local model with weak tool calling; support is experimental	Confirm with a hosted model, then try a larger local model with tool support
Test alert curl returns 410	Posting to /api/v1/alerts, removed in Alertmanager 0.27.0	Use /api/v2/alerts as in Step 7
holmes investigate alertmanager finds no alerts	Port-forward dropped, wrong service, or label filter too strict	Check curl http://localhost:9093/api/v2/alerts, drop --alertmanager-label and retry

FAQ

Do I need both HolmesGPT and K8sGPT?

No, but they complement each other. K8sGPT is a fast, cheap, deterministic scan that can run on a schedule with no model at all. HolmesGPT is the one you call when an alert fires and the cause may sit outside the object that is failing.

Can HolmesGPT or K8sGPT change my cluster?

With the read-only ServiceAccount from Step 1, no: the API server refuses any write. Without it, both act with your kubeconfig's permissions. HolmesGPT's built-in Kubernetes tools are read-only, but its bash toolset can run any command you approve at the prompt, and its optional Kubernetes Remediation MCP adds write tools that wait for human approval on every mutation.

Can I run this entirely offline with a local model?

K8sGPT works well with Ollama because the model only explains findings. HolmesGPT supports Ollama too, but its docs call the support experimental and warn about inconsistent tool calling, so expect weaker investigations than with a hosted model.

How much does a HolmesGPT investigation cost?

In the project's published 28 October 2026 benchmark run, average cost per scenario ranged from $0.04 to $0.20 depending on the model. Use --log-costs to see your own numbers, and lower max_steps to cap the worst case.

When I set this up on a cluster, the first thing I check is the auth can-i output from Step 1, because every other safety claim depends on it. After that I run K8sGPT without --explain to see what the rules alone catch, and only then let HolmesGPT loose on one alert route with --show-tool-output on, so I can see whether it is reasoning or pattern-matching. Official docs: HolmesGPT, HolmesGPT Alertmanager walkthrough, K8sGPT and Kubernetes RBAC.

Related: How to Fix Kubernetes Pods Stuck in Pending: Causes and Commands (2026), How to Reduce Alert Fatigue with Alertmanager: Step-by-Step (2026) and How to Fix Kubernetes CrashLoopBackOff: Causes and Commands (2026).