How to Benchmark LLM Inference: TTFT, Throughput and Goodput (2026)

This guide is for platform and ML engineers who have an LLM serving endpoint (vLLM, SGLang, TensorRT-LLM or anything OpenAI-compatible) and need numbers they can defend: how many users it can hold, at what time to first token, and where it falls over. At the end you will have a repeatable benchmark with a pinned workload, a concurrency sweep saved as JSON, a goodput figure against a stated SLO, and a short list of the ways benchmarks mislead. Flags are checked against the vLLM v0.32.0 docs and source and the GuideLLM v0.8.1 CLI, which I installed in a scratch virtualenv and ran against GuideLLM's built-in mock server. You need a GPU host running the server, Docker, and Python 3.11 or newer for the client tools.

If you are still choosing an engine, read the sibling post first: LLM Inference Engines 2026: vLLM vs SGLang vs TensorRT-LLM vs Dynamo. This guide is how you test that choice on your own traffic.

Step 1: Define the metrics and your SLO before you run anything

Most bad LLM benchmarks fail here, by reporting one tokens-per-second figure with no latency attached. Pick definitions, then targets.

• TTFT (time to first token): from sending the request to receiving the first streamed token. Dominated by queueing plus prefill, so it grows with prompt length and load. This is what users feel as "is it thinking".
• ITL (inter-token latency): the gap between consecutive streamed tokens, pooled across all tokens of all requests.
• TPOT (time per output token): per request, (end-to-end latency - TTFT) / (output tokens - 1). That is the formula in vLLM's benchmark source. TPOT is averaged per request, ITL per token, so long responses weigh more in ITL. They diverge when output lengths vary.
• E2EL (end-to-end latency): the whole request. What a non-streaming caller, such as an agent tool call, waits on.
• Throughput: output tokens per second and requests per second across the server. Say which one.
• Goodput: requests per second that met all your SLOs. The term comes from the DistServe paper, and it is the number that matters, because throughput you can only reach by blowing the latency target is not capacity.

Write the workload and SLO down in a file that lives next to the results. Everything later reads from it.

# bench.env: the contract for this benchmark. Commit it with the results.
export MODEL="Qwen/Qwen3-8B"          # served model, also the tokenizer
export BASE_URL="http://127.0.0.1:8000"
export ISL=1024                        # input sequence length (tokens)
export OSL=256                         # output sequence length (tokens)
export SLO_TTFT_MS=500                 # p99 target for interactive chat
export SLO_TPOT_MS=50                  # about 20 tokens/s per user
export CONCURRENCY="1 2 4 8 16 32 64 128"

Choose ISL and OSL from your logs, not from habit. A RAG endpoint with 6,000-token prompts and 200-token answers is a prefill benchmark; a coding agent producing 2,000 tokens is a decode benchmark. They have different knees.

Step 2: Start the server pinned, and record exactly what you tested

A benchmark result without the server version, flags and GPU is not reproducible, and engines change fast (vLLM shipped v0.30, v0.31 and v0.32 in four weeks, per its releases page). Pin the image and capture the environment.

source bench.env
mkdir -p results && cd results

# vLLM v0.32.0, one GPU. Pin the tag; never benchmark :latest.
docker run -d --name vllm-bench --gpus '"device=0"' --ipc=host \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -p 8000:8000 vllm/vllm-openai:v0.32.0 \
  --model "$MODEL" \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256

# Wait until it serves, then record the environment next to the results.
until curl -sf "$BASE_URL/v1/models" >/dev/null; do sleep 5; done
curl -s "$BASE_URL/version"            > server-version.json
docker inspect vllm-bench --format '{{.Config.Image}} {{.Args}}' > server-args.txt
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv > gpu.csv

Run nothing else on that GPU. A second process holding memory changes how much KV cache vLLM can allocate, which changes the batch size, which changes everything. If you deploy on Kubernetes, the same pinning applies to the manifests in How to Deploy vLLM on Kubernetes with GPUs.

Step 3: Smoke test with vllm bench serve and warm up

vllm bench serve ships in the same image and talks to any OpenAI-compatible endpoint, so you can point it at SGLang or TensorRT-LLM too. Run a small test first to check for errors and that output lengths are what you asked for.

source ../bench.env
docker run --rm --network host \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  --entrypoint vllm vllm/vllm-openai:v0.32.0 \
  bench serve \
    --backend openai --base-url "$BASE_URL" --endpoint /v1/completions \
    --model "$MODEL" \
    --dataset-name random \
    --random-input-len "$ISL" --random-output-len "$OSL" \
    --ignore-eos \
    --num-prompts 50 --num-warmups 10 \
    --max-concurrency 4

The summary block starts with Serving Benchmark Result and lists successful requests, request and token throughput, then mean, median and P99 for TTFT, TPOT and ITL (the vLLM benchmark docs show a full example). Check two things before going further: failed requests is zero, and total generated tokens is num-prompts × OSL. If it is lower, the model stopped early and you forgot --ignore-eos, or the server is truncating. --num-warmups matters because the first requests after startup pay for CUDA graph capture and kernel autotuning.

Step 4: Run a concurrency sweep and measure goodput

One load level tells you nothing about capacity. Sweep it. There are two ways to generate load, and you should know which you are using:

• Closed loop (--max-concurrency): N requests in flight; a new one starts when one finishes. It answers "how many simultaneous users", and it can never overload the server, so it hides queueing.
• Open loop (--request-rate): requests arrive at a rate (Poisson by default, --burstiness 1.0) whether or not the server keeps up. It answers "what happens at 20 requests per second", and it is the only way to see the queue blow up. The bench serve docs note the achieved rate may be lower than requested if the server cannot keep up.

Start with a closed-loop sweep to find the shape, then confirm the knee with an open-loop run.

source ../bench.env
for C in $CONCURRENCY; do
  N=$(( C * 10 > 200 ? C * 10 : 200 ))   # enough requests for a stable p99
  docker run --rm --network host -v "$PWD:/out" \
    -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
    --entrypoint vllm vllm/vllm-openai:v0.32.0 \
    bench serve \
      --backend openai --base-url "$BASE_URL" --endpoint /v1/completions \
      --model "$MODEL" --dataset-name random \
      --random-input-len "$ISL" --random-output-len "$OSL" --ignore-eos \
      --num-prompts "$N" --num-warmups 10 \
      --request-rate inf --max-concurrency "$C" \
      --percentile-metrics ttft,tpot,itl,e2el \
      --metric-percentiles 50,90,99 \
      --goodput ttft:"$SLO_TTFT_MS" tpot:"$SLO_TPOT_MS" \
      --save-result --result-dir /out --result-filename "sweep-c${C}.json" \
      --metadata engine=vllm-0.32.0 gpu="$(head -2 gpu.csv | tail -1 | cut -d, -f1)"
done

# Tabulate: concurrency, req/s, goodput req/s, output tok/s, p99 TTFT, p99 TPOT
for f in sweep-c*.json; do
  jq -r '[.max_concurrency, (.request_throughput*100|round/100),
          (.request_goodput*100|round/100), (.output_throughput|round),
          (.p99_ttft_ms|round), (.p99_tpot_ms*10|round/10)] | @tsv' "$f"
done | sort -n

The JSON keys (request_goodput, output_throughput, p99_ttft_ms and so on) are the ones vllm bench serve writes in v0.32.0. For an open-loop check around the knee, rerun the loop with --request-rate 2 4 8 ... values instead of concurrency, keeping --max-concurrency unset. If you want vLLM to drive server-flag sweeps for you as well, vllm bench sweep serve takes JSON files of serve and bench parameters and runs their Cartesian product, three runs each by default.

Step 5: Replace random prompts with real ones

Random token prompts are good for controlled comparisons and bad at predicting production. They have no shared prefixes (so prefix caching does nothing), fixed lengths (so batching is unrealistically tidy) and, with --ignore-eos, no natural stopping. Run the same sweep with at least one realistic dataset.

# Option A: a public conversational set (ShareGPT), real length distribution
wget -q https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Option B: your own prompts, exported from logs as JSONL with a "prompt" field
#   {"prompt": "Summarise this incident timeline: ..."}
head -c 300 prompts.jsonl

# Same sweep body as Step 4, with the dataset flags swapped:
# (both files sit in results/, which the container sees as /out)
#   --dataset-name sharegpt --dataset-path /out/ShareGPT_V3_unfiltered_cleaned_split.json
#   --dataset-name custom   --dataset-path /out/prompts.jsonl --custom-output-len 256

# Option C: measure prefix caching directly (system prompts, agent loops)
#   --dataset-name prefix_repetition \
#   --prefix-repetition-prefix-len 512 --prefix-repetition-suffix-len 128 \
#   --prefix-repetition-num-prefixes 5 --prefix-repetition-output-len 128

Scrub prompts exported from logs before they leave the production boundary. For agents, the prefix-repetition run is the one to compare engines on, because that is where SGLang's RadixAttention and vLLM's prefix cache differ.

Step 6: Cross-check with GuideLLM sweeps and goodput search

GuideLLM, also under the vLLM project, is a second, independent client. Version 0.7 replaced guidellm benchmark with guidellm run and typed kind=... options (v0.7.0 migration guide), so older blog posts using guidellm benchmark --rate-type no longer work, and v0.8.0 added SLO-based goodput. Its sweep profile runs a synchronous baseline, a maximum-throughput run, then constant rates in between, and its goodput profile searches for the highest load that meets an SLO.

python3 -m venv .venv && . .venv/bin/activate
pip install "guidellm[recommended]==0.8.1"

# Concurrency sweep, same workload shape as Step 4
guidellm run \
  --backend kind=openai_http,target="$BASE_URL" \
  --profile kind=concurrent --override profile.streams 1,4,16,64 \
  --constraint kind=max_duration,seconds=120 \
  --data kind=synthetic_text,prompt_tokens="$ISL",output_tokens="$OSL" \
  --tokenizer kind=hf_auto,model="$MODEL" \
  --output kind=json,path=guidellm/concurrent.json --output kind=console

# Highest load where 99% of requests meet TTFT and TPOT targets
guidellm run \
  --backend kind=openai_http,target="$BASE_URL" \
  --profile kind=goodput,target_attainment=0.99 \
  --data kind=synthetic_text,prompt_tokens="$ISL",output_tokens="$OSL" \
  --metrics "{\"kind\":\"generative\",\"slo\":{\"ttft_ms\":$SLO_TTFT_MS,\"tpot_ms\":$SLO_TPOT_MS}}" \
  --constraint kind=max_duration,seconds=120

To check the harness before burning GPU hours, GuideLLM includes guidellm mock-server, which fakes a model with configurable TTFT and ITL. I ran a smaller version of the concurrency sweep against it (1, 4, 8 and 16 streams, 256-token prompts, 128 output tokens, 80 ms TTFT, 12 ms ITL, 15 seconds per level) and extracted results with jq. This is real output:

$ jq -r '.benchmarks[] | [.config.strategy.streams,
    (.metrics.time_to_first_token_ms.successful.percentiles.p99|floor),
    (.metrics.inter_token_latency_ms.successful.percentiles.p99*10|floor/10),
    (.metrics.output_tokens_per_second.successful.mean|floor)] | @tsv' out/benchmarks.json
1	117	13.1	68
4	100	12.8	298
8	111	12.8	614
16	105	12.7	1228

Note what is wrong with it: throughput scaled almost exactly linearly with concurrency and latency did not move. That is what a mock with no compute limit does, and it is the signature to look for in a real run too. If your GPU results look like this, you are not loading the GPU: the client, network or a proxy is the bottleneck. The mock run also reported about 433 input tokens per request against the 256 I asked for, because the mock counts with its own tokenizer, not the gpt2 tokenizer I gave the client. On a real server, always pass the served model's tokenizer.

Step 7: Read the results and find the knee

Plot p99 TTFT and output tokens per second against concurrency. On a real GPU you will see three regions: throughput climbing near-linearly while latency is flat; a knee where throughput flattens and TTFT starts rising; and saturation, where throughput stays flat and TTFT grows with the queue. Your capacity is the highest concurrency before p99 TTFT or TPOT crosses the SLO, which is what the goodput column already tells you. Plan to run at 70 to 80 percent of it, so a burst has somewhere to go.

Then confirm on the server side that the client saw what the server did. vLLM exposes Prometheus metrics on /metrics; the names below are from the vLLM metrics design doc.

# Queue depth and KV cache pressure during the run
curl -s "$BASE_URL/metrics" | grep -E '^vllm:(num_requests_running|num_requests_waiting|kv_cache_usage_perc)'

# With Prometheus scraping the server, p99 TTFT as the server measured it:
#   histogram_quantile(0.99, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[1m])))
# Time spent queued before scheduling (the part of TTFT that is load, not prefill):
#   histogram_quantile(0.99, sum by (le) (rate(vllm:request_queue_time_seconds_bucket[1m])))

If num_requests_waiting is above zero at the knee while kv_cache_usage_perc is near 1.0, you are KV-cache bound: a shorter --max-model-len, an FP8 KV cache or more GPUs will move the knee. If the queue grows while the KV cache is half empty, you are compute bound or capped by --max-num-seqs. Client and server p99 TTFT should agree within network latency; a large gap means the client is the bottleneck.

Troubleshooting

Symptom	Likely cause	Fix
Throughput scales linearly, latency never moves	Not loading the GPU: client-bound, or concurrency far below the knee	Raise concurrency until TTFT rises; run the client on a separate host with enough CPU
Generated tokens far below num-prompts × OSL	Model hits EOS early on random prompts	Add --ignore-eos; check the server does not cap max_tokens
First run is much slower than the second	CUDA graph capture, autotuning, cold caches	Use --num-warmups, discard the first run, report the median of three
Huge TTFT but low ITL at every level	Open-loop rate above capacity, requests queueing	Lower --request-rate; check vllm:request_queue_time_seconds
Engine A beats B by 3x on random prompts	Different effective settings: max sequences, KV dtype, context length	Match --max-model-len, batch caps and quantisation; compare via goodput, not peak tok/s
Results change run to run by over 10%	Too few requests for stable p99, noisy neighbour, thermal throttling	At least 200 requests per level, dedicated GPU, check clocks with nvidia-smi -q -d CLOCK
Input token counts differ from requested	Client tokenizer differs from the server's, or chat template added	Pass the served model as tokenizer; use --skip-chat-template if prompts already include one
Prefix caching shows no gain	Unique random prompts share nothing	Use prefix_repetition or real multi-turn traces
Many failed requests at high concurrency	Timeouts, server request limit, proxy connection cap	Check server logs and proxy limits; benchmark direct to the pod first

FAQ

What is the difference between TTFT, TPOT and ITL?

TTFT is the wait until the first token, mostly queueing plus prefill. TPOT is a per-request average of the time between output tokens after the first; ITL is the same gap measured per token across all requests. They match when output lengths are similar and diverge when some responses are much longer.

Should I benchmark with request rate or max concurrency?

Use both. A --max-concurrency sweep tells you how many simultaneous streams the server sustains and finds the knee quickly. A --request-rate run at and just above that point shows what happens when arrivals exceed capacity, which closed-loop tests cannot show.

Is GenAI-Perf still the NVIDIA benchmarking tool?

No. The GenAI-Perf README says it is being phased out with no new features, and points to AIPerf, whose latest release is v0.13.0 from September 2026. For vLLM and SGLang, vllm bench serve and GuideLLM are the most direct choices.

How many requests do I need per concurrency level?

Enough that the p99 is not decided by one or two requests. I use at least 200, or ten times the concurrency, whichever is larger, and repeat each level three times. If the three runs disagree by more than about 10 percent, fix the environment before trusting any of them.

In production I check two things first: that generated token counts match what I asked for, and that the server's own /metrics TTFT agrees with the client's. Most misleading benchmarks fail one of those. After that I keep a short sweep (four concurrency levels, random plus one real dataset) in the release pipeline, so an engine or flag change has to beat the last run's goodput before it ships. The official references: vllm bench serve CLI, vLLM benchmark guide, GuideLLM and the vLLM metrics reference.

Related: SRE for LLM Apps in 2026: SLOs for TTFT, Provider Outages and Evals, How to Deploy vLLM on Kubernetes with GPUs: Step-by-Step (2026) and LLM Inference in 2026: Prefill Is a Different Machine Now, and the KV Cache Has a Disk Tier.