How to Install NVIDIA GPU Operator and Share GPUs: Time-Slicing, MIG (2026)

This guide is for platform engineers who have a Kubernetes cluster with NVIDIA GPU nodes and want them working properly: drivers installed and upgraded by the cluster, a verified nvidia.com/gpu resource, and more than one workload per card where that makes sense. By the end you will have the NVIDIA GPU Operator v26.7.2 installed with Helm, a passing CUDA test pod, time-slicing configured from a ConfigMap, MIG partitions on Ampere-or-newer cards, and DCGM metrics in Prometheus. Versions are pinned to what was current on 31 October 2026: GPU Operator v26.7.2 (released 14 October), which ships driver 595.91.07, device plugin v0.20.1, MIG Manager v0.15.1 and DCGM Exporter 4.6.1-4.8.4, validated on Kubernetes 1.33 to 1.37. You need kubectl and Helm 3 or later, cluster-admin rights, and at least one GPU node running a supported OS. If you want the background first (why the scheduler only sees an integer, and how this compares with DRA), read the companion explainer, GPUs on Kubernetes Explained (2026).

Step 1: Check the GPU nodes before you install anything

Most failed installs are decided before Helm runs. Three things to check on each GPU node.

First, is a driver already there? EKS accelerated AMIs, GKE's default node pools and AKS GPU node pools preinstall one. The Operator's driver pod will detect a host driver and stand down, per the getting started page, but it is cleaner to set driver.enabled=false deliberately. On GKE and AKS you usually want to turn off the managed driver instead (the explainer's table has the flags).

Second, is nouveau loaded? The open-source driver blocks the NVIDIA kernel module, and the troubleshooting page lists it as the cause of driver pods stuck in init.

Third, does the cluster enforce Pod Security Admission? Every Operator component runs privileged, so the namespace needs the privileged label.

# On a GPU node (SSH or a debug pod with host access)
lspci | grep -i nvidia             # the card is visible on the PCI bus
nvidia-smi 2>/dev/null || echo "no host driver"   # a host driver means driver.enabled=false
lsmod | grep -i nouveau            # any output here must be fixed first

# From your workstation
kubectl get nodes -L node.kubernetes.io/instance-type
kubectl describe node <gpu-node> | grep -A5 Taints   # note GPU taints for later

# Namespace with the Pod Security label the Operator needs
kubectl create namespace gpu-operator
kubectl label --overwrite namespace gpu-operator \
  pod-security.kubernetes.io/enforce=privileged

If your GPU nodes carry a taint, check that the Operator's DaemonSets tolerate it. The v26.7.2 chart's daemonsets.tolerations value already tolerates nvidia.com/gpu with effect NoSchedule, so the common nvidia.com/gpu=present:NoSchedule taint is covered; any other taint key needs adding to that list, or the DaemonSets show DESIRED 0 on those nodes and nothing is scheduled.

Step 2: Install the GPU Operator v26.7.2 with Helm

Pin the chart version. An unpinned install picks up whatever NVIDIA published last, and a new chart can move the driver branch underneath your CUDA images. I keep the values in Git next to the cluster config rather than as a pile of --set flags.

# values-gpu-operator.yaml for chart nvidia/gpu-operator v26.7.2
# Defaults checked against deployments/gpu-operator/values.yaml at tag v26.7.2
driver:
  enabled: true          # set false if the node image already has a driver
  version: "595.91.07"   # chart default; write it down next to your CUDA versions
toolkit:
  enabled: true          # set false if nvidia-container-toolkit is preinstalled
cdi:
  enabled: true          # default since v25.10.0: GPUs injected via CDI
mig:
  strategy: single       # single or mixed; see Step 5
dcgmExporter:
  enabled: true
  serviceMonitor:
    enabled: true        # default true; needs the Prometheus Operator CRDs
    additionalLabels:
      release: kube-prometheus-stack   # match your Prometheus serviceMonitorSelector

Install from NVIDIA's Helm repository:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --version v26.7.2 \
  --values values-gpu-operator.yaml \
  --wait

From the 26.7 line the chart is also published as an OCI artifact, signed with NVIDIA's key. If you prefer that path, verify it with Cosign first and install from oci://nvcr.io/nvidia/cloud-native-charts/gpu-operator:

cosign verify --key https://api.ngc.nvidia.com/v2/catalog/containers/public-key \
  --insecure-ignore-tlog \
  nvcr.io/nvidia/cloud-native-charts/gpu-operator:v26.7.2

helm install gpu-operator oci://nvcr.io/nvidia/cloud-native-charts/gpu-operator \
  --namespace gpu-operator --version v26.7.2 \
  --values values-gpu-operator.yaml --wait

The driver container compiles or loads a kernel module on each node, so the first install takes several minutes per node. --wait only waits for the Operator Deployment, not for the drivers, so do not read a green Helm result as "GPUs ready".

Step 3: Verify the install with nvidia-smi and a CUDA test pod

Watch the namespace until everything is Running or Completed. The pod list in NVIDIA's docs for a default install looks like this (suffixes will differ):

kubectl get pods -n gpu-operator
# gpu-feature-discovery-xxxxx                    1/1  Running
# gpu-operator-xxxxxxxxxx-xxxxx                  1/1  Running
# gpu-operator-node-feature-discovery-gc-...     1/1  Running
# gpu-operator-node-feature-discovery-master-... 1/1  Running
# gpu-operator-node-feature-discovery-worker-... 1/1  Running
# nvidia-container-toolkit-daemonset-xxxxx       1/1  Running
# nvidia-cuda-validator-xxxxx                    0/1  Completed
# nvidia-dcgm-exporter-xxxxx                     1/1  Running
# nvidia-device-plugin-daemonset-xxxxx           1/1  Running
# nvidia-driver-daemonset-xxxxx                  1/1  Running
# nvidia-operator-validator-xxxxx                1/1  Running

kubectl get clusterpolicies.nvidia.com
# NAME             STATUS   AGE
# cluster-policy   ready    6m

Then check the driver from inside the driver container, and confirm the node now advertises the resource:

kubectl exec -n gpu-operator ds/nvidia-driver-daemonset -c nvidia-driver-ctr -- nvidia-smi
# Shows driver 595.91.07, the CUDA version it supports, and each GPU

kubectl get node <gpu-node> -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
# 1   (or the number of cards on the node)

kubectl get node <gpu-node> --show-labels | tr ',' '\n' | grep nvidia.com/gpu.product
# nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3   (written by GPU Feature Discovery)

Finally, prove a real workload can get a GPU. Two pods: one that runs nvidia-smi the way an application container would see it, and NVIDIA's vector-add sample.

# gpu-test.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi
spec:
  restartPolicy: Never
  containers:
  - name: nvidia-smi
    image: nvidia/cuda:13.0.2-base-ubuntu24.04
    command: ["nvidia-smi", "-L"]
    resources:
      limits:
        nvidia.com/gpu: 1      # GPUs go in limits; requests, if set, must match
---
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vectoradd
    image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
    resources:
      limits:
        nvidia.com/gpu: 1

kubectl apply -f gpu-test.yaml
kubectl logs pod/nvidia-smi
# GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-...)
kubectl logs pod/cuda-vectoradd
# [Vector addition of 50000 elements]
# ...
# Test PASSED
# Done
kubectl delete -f gpu-test.yaml

If both pass, the driver, the container toolkit, CDI injection and the device plugin all work. Add a GPU toleration to the test pods if your nodes are tainted.

Step 4: Share a GPU with time-slicing

Time-slicing tells the device plugin to advertise each physical GPU as several nvidia.com/gpu units. CUDA interleaves the processes. Be clear about what you are buying: the device plugin README says each workload "runs in the same fault-domain" as the others, with no memory limits, so one pod that allocates all the memory starves or kills the rest. It suits development, notebooks and small batch jobs from one team.

Create the configuration as a ConfigMap in the Operator's namespace. The structure follows NVIDIA's GPU sharing docs; I have turned on failRequestsGreaterThanOne, which the README recommends but leaves off by default, so a pod asking for two slices fails loudly instead of silently getting no more compute than one.

# time-slicing-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false          # true advertises nvidia.com/gpu.shared instead
        failRequestsGreaterThanOne: true
        resources:
        - name: nvidia.com/gpu
          replicas: 4                   # each card appears as 4 schedulable units
  tesla-t4: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        failRequestsGreaterThanOne: true
        resources:
        - name: nvidia.com/gpu
          replicas: 2

Apply it and point the ClusterPolicy at it. default: any applies the any profile to every GPU node; individual nodes can opt into another key with a label.

kubectl apply -f time-slicing-config.yaml

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  -n gpu-operator --type merge \
  -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config", "default": "any"}}}}'

# Optional: give T4 nodes the tesla-t4 profile instead
kubectl label node --selector=nvidia.com/gpu.product=Tesla-T4 \
  nvidia.com/device-plugin.config=tesla-t4 --overwrite

The Operator does not watch these ConfigMaps for edits. After any later change, restart the plugin:

kubectl rollout restart -n gpu-operator daemonset/nvidia-device-plugin-daemonset

Verify that capacity has multiplied and the product label has the -SHARED suffix:

kubectl describe node <gpu-node> | grep -E 'nvidia.com/gpu(\.replicas|\.product)?[:=]'
# nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3-SHARED
# nvidia.com/gpu.replicas=4
# nvidia.com/gpu:     4      (Capacity, for one physical card)
# nvidia.com/gpu:     4      (Allocatable)

Two side effects to plan for. Quotas and cost reports that count nvidia.com/gpu now count slices, not cards. And DCGM Exporter cannot attribute metrics to containers while time-slicing is on, so per-pod GPU dashboards stop working on those nodes; node-level metrics are still fine.

Step 5: Partition GPUs with MIG on A100, H100 and newer

Multi-Instance GPU splits a card in hardware: each instance has its own memory, cache and streaming multiprocessors, so a crash or memory blow-up stays inside its slice. The Operator's MIG Manager applies a named geometry when you label a node, per the MIG docs.

Pick a strategy first. With single, every GPU on the node uses the same profile and slices are still advertised as nvidia.com/gpu, so existing manifests keep working. With mixed, each profile becomes its own resource such as nvidia.com/mig-1g.10gb or nvidia.com/mig-3g.40gb, which you need for any geometry with more than one profile.

# Switch strategy on a running install (or set mig.strategy in the values file)
kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
  -p='[{"op":"replace", "path":"/spec/mig/strategy", "value":"single"}]'

# MIG Manager stops every GPU pod on the node it reconfigures, so drain it first
kubectl cordon <gpu-node>
kubectl drain <gpu-node> --ignore-daemonsets --delete-emptydir-data

# Seven 1g.10gb instances per H100 80GB
kubectl label node <gpu-node> nvidia.com/mig.config=all-1g.10gb --overwrite

# Wait until the state label reads success
kubectl get node <gpu-node> -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}{"\n"}'
# success

kubectl exec -n gpu-operator ds/nvidia-driver-daemonset -c nvidia-driver-ctr -- nvidia-smi -L
# GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-...)
#   MIG 1g.10gb     Device  0: (UUID: MIG-...)
#   ... seven devices in total

kubectl uncordon <gpu-node>

On some cloud instances the change needs a reboot; enable it with --set migManager.env[0].name=WITH_REBOOT --set-string migManager.env[0].value=true and the state label will show rebooting in between. If the node has a host-installed driver, MIG Manager also has to stop host GPU services, which are listed in the default-gpu-clients ConfigMap.

For a geometry the built-in profiles do not offer, write your own. The ConfigMap must have a key named config.yaml, and because this one mixes two profiles the strategy must be mixed:

# custom-mig-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-mig-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-disabled:
        - devices: all
          mig-enabled: false
      five-1g-one-2g:                 # 5 x 1g.10gb + 1 x 2g.20gb on an H100 80GB
        - devices: all
          mig-enabled: true
          mig-devices:
            "1g.10gb": 5
            "2g.20gb": 1

kubectl apply -f custom-mig-config.yaml
kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
  -p='[{"op":"replace", "path":"/spec/migManager/config/name", "value":"custom-mig-config"}]'
kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
  -p='[{"op":"replace", "path":"/spec/mig/strategy", "value":"mixed"}]'
kubectl label node <gpu-node> nvidia.com/mig.config=five-1g-one-2g --overwrite

A workload on a mixed node then asks for the slice by name:

resources:
  limits:
    nvidia.com/mig-1g.10gb: 1   # mixed strategy resource name

You can time-slice MIG slices too: the device plugin accepts nvidia.com/mig-1g.10gb and the other mixed-strategy names in the timeSlicing.resources list. To turn MIG off again, label the node nvidia.com/mig.config=all-disabled.

Step 6: Scrape DCGM Exporter metrics with Prometheus

The Operator runs DCGM Exporter as a DaemonSet behind a Service called nvidia-dcgm-exporter on port 9400 (gpu-metrics), as defined in the Operator's service manifest. In chart v26.7.2 the dcgmExporter.serviceMonitor.enabled value defaults to true, so a cluster with the Prometheus Operator only needs the ServiceMonitor's labels to match its selector; the additionalLabels line in Step 2 handles the common kube-prometheus-stack case. If you have not set up that stack yet, the kube-prometheus-stack guide covers it.

Check the endpoint directly first:

kubectl port-forward -n gpu-operator svc/nvidia-dcgm-exporter 9400:9400 &
curl -s localhost:9400/metrics | grep -E '^DCGM_FI_(DEV_GPU_UTIL|DEV_FB_USED|PROF_GR_ENGINE_ACTIVE)' | head
# DCGM_FI_DEV_GPU_UTIL{gpu="0",...,pod="vllm-0",namespace="ml"} 87

The default counter set includes these, and they are the ones I put on the first dashboard:

# Busy percentage per GPU (the nvidia-smi number)
avg by (Hostname, gpu) (DCGM_FI_DEV_GPU_UTIL)

# Fraction of time the graphics engine is actually active (0 to 1); the honest one
avg by (Hostname, gpu) (DCGM_FI_PROF_GR_ENGINE_ACTIVE)

# Tensor core activity: low here on a "busy" card means it is not doing matmul work
avg by (Hostname, gpu) (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE)

# Framebuffer memory used per pod, in MiB (empty on time-sliced nodes)
sum by (namespace, pod) (DCGM_FI_DEV_FB_USED)

# Alert input: last Xid error on each GPU (non-zero deserves a look)
max by (Hostname, gpu) (DCGM_FI_DEV_XID_ERRORS) > 0

Troubleshooting

Symptom	Likely cause	Fix
Operand pods stuck in Init:0/1, driver pod restarting	Driver container failing to build or load the module	kubectl logs -n gpu-operator <driver-pod> -c nvidia-driver-ctr and -c k8s-driver-manager; check dmesg for NVRM errors and that the pod can reach package mirrors
Driver fails to initialize the GPU, system journal shows Failed to enable MSI-X	nouveau module is loaded	Blacklist nouveau, rebuild initramfs, reboot the node (commands in NVIDIA's troubleshooting page)
Node has no nvidia.com/gpu in Allocatable	Device plugin not running yet, or driver failed earlier in the chain	Fix the first failing pod in order: NFD, driver, toolkit, device plugin. kubectl get pods -n gpu-operator -o wide for that node
kubectl get ds -n gpu-operator shows DESIRED 0	Node taints with no matching tolerations	Remove the taint or add it to daemonsets.tolerations in the values file and run helm upgrade
Validator: no runtime for "nvidia" is configured	Toolkit never configured containerd, usually because the driver is not up	Fix the driver first, then check the toolkit pod logs and containerd config dump on the node
Node advertises 7 GPUs instead of 8	Device plugin marked a GPU unhealthy after an Xid error	sudo dmesg | grep -i xid; look for "marking device as unhealthy" in plugin logs; Xid 79 means hardware or driver fault
Time-slicing ConfigMap edited but capacity unchanged	Operator does not watch the ConfigMap	kubectl rollout restart -n gpu-operator daemonset/nvidia-device-plugin-daemonset
Pod fails with UnexpectedAdmissionError	failRequestsGreaterThanOne: true and the pod asked for more than one shared GPU	Request nvidia.com/gpu: 1 on time-sliced nodes, or schedule onto a non-shared pool
nvidia.com/mig.config.state never reaches success	GPU pods or host GPU processes still running	Drain the node, check kubectl logs -n gpu-operator -l app=nvidia-mig-manager -c nvidia-mig-manager, enable WITH_REBOOT on cloud instances
DCGM dashboards empty for pods but node metrics fine	Time-slicing is enabled on that node	Expected limitation; use MIG where per-pod attribution matters

When a GPU pod itself is the thing stuck, the generic causes still apply: see How to Fix Kubernetes Pods Stuck in Pending and How to Fix Kubernetes CrashLoopBackOff.

FAQ

Do I need the GPU Operator on EKS, GKE or AKS?

No. All three can install drivers and a device plugin for you, and EKS Auto Mode manages the whole stack. The Operator is worth it when you need a specific driver branch, MIG Manager, NVIDIA's time-slicing configuration or the same DCGM metrics on every cloud.

Should I use time-slicing or MIG?

Use MIG when workloads belong to different teams or customers, or when one pod's memory use must not affect another, because MIG partitions memory and compute in hardware. Use time-slicing for development, notebooks and light batch jobs from one team, on any NVIDIA GPU. They combine: you can time-slice individual MIG slices.

Does a time-sliced GPU give each pod a quarter of the memory?

No. Time-slicing shares compute time only. Every pod sees the full card's memory and can allocate all of it, which is why one runaway process can take down its neighbours. MPS (experimental in the device plugin) and MIG are the options that cap memory per workload.

How do I upgrade the GPU Operator safely?

Bump the pinned chart version in Git, read that release's notes for driver branch changes, and run helm upgrade with the same values file. Driver upgrades roll node by node; nodes that fail show nvidia.com/gpu-driver-upgrade-state=upgrade-failed, and you can retry one by relabelling it upgrade-required.

When a GPU node misbehaves, the first thing I check is not the GPU: it is kubectl get pods -n gpu-operator -o wide for that node, read left to right in the order of the figure above, because every component waits on the one before it and the first non-running pod is almost always the real fault. In production I pin the chart and driver in Git, keep time-slicing on clearly labelled development pools only, and put DCGM_FI_PROF_GR_ENGINE_ACTIVE next to allocation numbers before anyone asks for more cards. If you are serving models on these nodes next, How to Deploy vLLM on Kubernetes with GPUs picks up from here. Official references: the GPU Operator getting started guide, GPU sharing (time-slicing), MIG support, troubleshooting and the Kubernetes Schedule GPUs page.

Related: GPUs on Kubernetes Explained (2026), How to Deploy vLLM on Kubernetes with GPUs: Step-by-Step (2026) and Kubernetes Cost Optimization in 2026.