How to Troubleshoot Linux Performance in 60 Seconds (2026 Checklist)

This guide is for anyone who has just SSHed into a slow Linux box (an EC2 instance, a Kubernetes node, a bare-metal database host) and needs to know within a minute whether the problem is CPU, memory, disk, network or the application. By the end you will have a repeatable checklist, the columns that matter in each tool's output, and a set of eBPF follow-up commands for when the counters point somewhere but do not explain why. It follows Brendan Gregg's Linux Performance Analysis in 60,000 Milliseconds, first published on the Netflix Tech Blog in November 2015, updated for what changed since: sysstat 12.8.0 renamed and removed iostat fields, free grew an available column, the kernel exposes pressure stall information, and bpftrace is at v0.27.0. You need root or sudo, and a kernel with BTF (most 5.10+ distribution kernels) for the eBPF steps.

Step 0: Install the tools before you need them

Half of the 60-second checklist lives in the sysstat package, and a minimal cloud image rarely has it. Bake these into your node images now, not during the incident.

# Debian / Ubuntu
sudo apt-get update
sudo apt-get install -y sysstat procps bpfcc-tools bpftrace linux-tools-common linux-tools-$(uname -r)

# RHEL / Rocky / Alma / Fedora
sudo dnf install -y sysstat procps-ng bcc-tools bpftrace bpftool

# Distribution bpftrace is often several releases behind. Pin the upstream static build instead:
curl -fsSLo bpftrace \
  https://github.com/bpftrace/bpftrace/releases/download/v0.27.0/bpftrace-x86_64
sudo install -m 0755 bpftrace /usr/local/bin/bpftrace   # use bpftrace-aarch64 on arm64
bpftrace --version   # expect: bpftrace v0.27.0

# eBPF tools need kernel BTF; this file must exist
ls /sys/kernel/btf/vmlinux

On Ubuntu the BCC tools are installed with a -bpfcc suffix (runqlat-bpfcc); on RHEL they live in /usr/share/bcc/tools/. The rest of this guide uses the bare names.

Step 1: The ten commands, in order

The checklist is ten commands. Run them in this order, one terminal, and copy the output into your incident notes as you go:

uptime
dmesg -T | tail          # -T prints human-readable timestamps
vmstat 1                 # Ctrl-C after 5-10 lines
mpstat -P ALL 1
pidstat 1
iostat -xz 1
free -m
sar -n DEV 1
sar -n TCP,ETCP 1
top

The idea, in Gregg's words, is to look for errors and saturation first, because they are easy to interpret, and only then at utilisation. Every resource you exonerate narrows the search. The diagram below is the order I follow.

Step 2: Read uptime and dmesg for context

$ uptime
 14:02:11 up 12 days,  3:41,  1 user,  load average: 30.02, 26.43, 19.02

The three numbers are exponentially damped averages over 1, 5 and 15 minutes. On Linux they count tasks wanting CPU and tasks in uninterruptible sleep (usually disk I/O), so a high load average is a demand signal, not a CPU signal. Compare the 1-minute and 15-minute figures: rising means the problem is happening now; a 1-minute value well below the 15-minute value means you may have arrived after it ended. Compare against the CPU count from nproc.

dmesg -T | tail -n 20
# Look for: "Out of memory: Killed process", "blocked for more than 120 seconds",
# "I/O error", "nf_conntrack: table full, dropping packet", "Possible SYN flooding"

Do not skip this one. An OOM kill or a disk error in dmesg often ends the investigation in ten seconds. If dmesg returns "Operation not permitted", use sudo or journalctl -k -n 50.

Step 3: Check CPU saturation with vmstat, mpstat and pidstat

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

Skip the first line, which shows averages since boot. Per the vmstat man page:

• r is runnable processes, running or waiting for a CPU. Sustained r greater than the CPU count is CPU saturation, and unlike load average it excludes I/O.
• b is processes blocked on I/O. Non-zero and growing points at disk or NFS.
• si/so are swap-in and swap-out. Anything consistently non-zero means memory pressure.
• us/sy/id/wa/st split CPU time. High us is application work; sy above about 20 percent is worth a look (Gregg's threshold); steady wa means CPUs are idle because tasks wait on disk; non-zero st on a VM means the hypervisor is taking cycles you are paying for.

Then look per CPU and per process:

mpstat -P ALL 1 3     # one CPU at ~100% %usr while others idle = single-threaded bottleneck
pidstat 1 5           # rolling per-process %CPU; 1590% means ~16 CPUs' worth for that PID
pidstat -t -p <PID> 1 # per-thread breakdown once you have a suspect

%soft climbing on a few CPUs in mpstat usually means network interrupt processing is concentrated on those cores, which is an RSS/IRQ affinity question rather than an application one.

Step 4: Check disk latency with iostat -xz

$ iostat -xz 1 3
Device  r/s  rkB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s  wkB/s  ...  w_await  wareq-sz  ...  aqu-sz  %util

-x gives extended stats, -z hides idle devices. Field names changed in sysstat 12.x, so older blog posts will confuse you. From the current iostat man page and changelog:

• r_await / w_await: average milliseconds per request including queue time. This is what the application feels. Compare to what the device should do (sub-millisecond for local NVMe, single-digit milliseconds for most network block storage).
• aqu-sz: average queue length, formerly avgqu-sz. Above 1 can indicate saturation, though parallel devices can handle more.
• rareq-sz / wareq-sz: average request size in kibibytes, replacing avgrq-sz (which was in sectors).
• %util: busy time. Close to 100 percent means saturation only for devices that serve requests serially. For RAID, modern SSDs and cloud volumes the man page says outright that it does not reflect their limits, so trust await and aqu-sz instead.
• svctm is gone. sysstat removed it because it was never accurate; ignore any guide that tells you to read it.

On cloud volumes, also check the provider's throughput and IOPS limits. A volume that hits its provisioned IOPS looks exactly like a slow disk.

Step 5: Check memory with free and pressure stall information

$ free -m
               total        used        free      shared  buff/cache   available
Mem:           ...

Ignore the free column. Linux uses idle memory for page cache and gives it back on demand, which is why the old -/+ buffers/cache line existed. The column that matters now is available, which the free man page defines as an estimate of memory available for new applications without swapping. If available is a small fraction of total and buff/cache is near zero, the box is short of memory and disk reads will rise because nothing stays cached.

The modern addition to this step is PSI, which reports how much time tasks spent stalled on a resource:

cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
# some avg10=0.00 avg60=0.00 avg300=0.00 total=0
# full avg10=0.00 avg60=0.00 avg300=0.00 total=0

# For a container or systemd service, read its cgroup v2 file instead:
cat /sys/fs/cgroup/system.slice/nginx.service/memory.pressure

Per the kernel PSI docs, some is the share of time at least one task was stalled and full is the share of time all non-idle tasks were stalled at once. Memory full avg10 above zero means real work is being lost to reclaim, and it fires before the OOM killer does. In Kubernetes this is the number to check before blaming a pod's memory limit; the OOMKilled guide covers the limit side.

Step 6: Check the network with sar

sar -n DEV 1 3        # rxkB/s, txkB/s per interface; compare to the NIC or instance bandwidth limit
sar -n TCP,ETCP 1 3   # active/s (outbound connects), passive/s (accepts), retrans/s

rxkB/s multiplied by 8 gives kilobits; compare it with your instance type's documented bandwidth, not the NIC's marketing number. retrans/s above zero on a LAN is a red flag: either packet loss or a host dropping packets because it is overloaded. A spike in active/s often means a client has lost connection pooling. These sar commands report live and do not need the sysstat collection service enabled.

Finish with top (or htop) to confirm nothing changed while you were reading. Press 1 for per-CPU lines and P/M to sort by CPU or memory.

Step 7: Apply the USE method to whatever is left

If the ten commands did not identify a clear culprit, switch from a checklist to a method. Gregg's USE method is one sentence: for every resource, check utilisation, saturation and errors. Make a small table for CPUs, memory, each disk, each NIC, and the less obvious ones (file descriptors, conntrack entries, the CPU cgroup quota), and fill in all three columns. Two cheap checks that often finish the table:

# Container CPU throttling (cgroup v2): nr_throttled and throttled_usec climbing = quota too low
cat /sys/fs/cgroup/<path-to-cgroup>/cpu.stat

# Conntrack table close to full causes silent drops
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

CPU throttling is the classic "CPU is 40 percent but latency is terrible" case on Kubernetes, and the requests and limits explainer goes into why.

Step 8: Drill down with BCC and bpftrace

Counters tell you which resource; eBPF tells you why. (If you want the background on how these tools load safely into the kernel, read the companion eBPF explainer.) These BCC tools come from iovisor/bcc v0.37.0; flags are from each tool's built-in examples.

# CPU: how long do runnable tasks wait for a CPU? (scheduler queue latency histogram)
sudo runqlat 1 5

# CPU: sample on-CPU stacks at 99 Hz for 10 s in folded format for a flame graph
sudo profile -F 99 -f 10 > /tmp/oncpu.folded

# Short-lived processes that never show up in top (crash-looping scripts, cron storms)
sudo execsnoop -T

# Disk: block I/O latency histogram per device
sudo biolatency -D 1 5

# Files: which opens are failing (missing config, permission errors)
sudo opensnoop -x -d 10

# Network: every TCP retransmit with addresses and state
sudo tcpretrans

# Nothing saturated but still slow? Where are threads blocked off-CPU, for 10 s
sudo offcputime -p <PID> -f 10 > /tmp/offcpu.folded

And the bpftrace one-liners worth memorising, adapted from the bpftrace one-liner tutorial (v0.27 syntax uses args.field; bpftrace 0.19 changed it from the older args->field):

# Syscall counts by process name (who is hammering the kernel?)
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Files opened, by process
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args.filename)); }'

# Block I/O request size distribution
sudo bpftrace -e 'tracepoint:block:block_rq_issue { @bytes = hist(args.bytes); }'

# On-CPU kernel stacks at 99 Hz
sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'

bpftrace also ships ready-made scripts (runqlat.bt, biolatency.bt, tcpretrans.bt, oomkill.bt) in its tools directory, useful where BCC is not installed.

Troubleshooting

Symptom	Likely cause	Fix
mpstat: command not found (also sar, iostat, pidstat)	sysstat not installed	apt-get install sysstat or dnf install sysstat; bake it into the image
High load average but low CPU in vmstat	Tasks in uninterruptible I/O (b column, wa)	Go to iostat -xz; check NFS mounts and dmesg for hung tasks
%util at 100% on an SSD or cloud volume, await normal	%util does not reflect limits for parallel devices	Judge by r_await/w_await and aqu-sz, and the volume's IOPS quota
vmstat si/so non-zero, available low	Memory exhaustion, swapping	Find the consumer with pidstat -r 1 or top sorted by M; fix limits or leak
Pod slow, node CPU fine	CFS quota throttling	Check cpu.stat nr_throttled; raise or remove the CPU limit
sar -n ETCP shows retrans/s above zero	Packet loss or overloaded peer	tcpretrans to find which peers; check NIC drops with ip -s link
BCC tool fails with missing kernel headers or BTF	No /sys/kernel/btf/vmlinux and no headers	Install linux-headers-$(uname -r) or use a BTF-enabled kernel
bpftrace error on args->filename	Syntax changed in bpftrace 0.19	Use args.filename
dmesg: read kernel buffer failed: Operation not permitted	kernel.dmesg_restrict=1	Use sudo dmesg or journalctl -k
Steal time (st) above a few percent	Noisy neighbour or burstable instance out of credits	Check CPU credit balance; move to a non-burstable instance type

FAQ

What is a good load average on Linux?

There is no universal number. Divide the load average by the CPU count from nproc: sustained values above 1.0 per CPU mean demand exceeds capacity, but Linux load includes tasks waiting on disk, so confirm with the vmstat r column and mpstat before calling it a CPU problem.

Why does Linux show almost no free memory?

Because Linux uses idle memory for the page cache and returns it when applications need it. Read the available column of free -m, not free, and confirm real pressure with vmstat si/so and /proc/pressure/memory.

Is it safe to run bpftrace and BCC on a production server?

Generally yes: the kernel verifier ensures the programs cannot crash the kernel. Overhead depends on how often the probe fires, so prefer tracepoints over kprobes on hot functions, set a duration, and avoid printing per-event output on very busy events.

Do I still need the 60-second checklist if I have Prometheus or Datadog?

Yes. Dashboards show aggregates at 15 to 60 second resolution and often miss per-CPU imbalance, per-process detail, short-lived processes and kernel messages. The checklist takes a minute and frequently answers what the dashboard only hints at.

When I get paged for a slow host, the first two things I read are dmesg and the vmstat r and wa columns, because between them they rule out or confirm the three most common causes (OOM, CPU saturation, disk waits) in under twenty seconds. On every node image I build, sysstat, bpftrace and the BCC tools are preinstalled and BTF is checked at build time, because installing packages on a struggling machine is the worst moment to do it. Official references: the sysstat documentation, the kernel PSI docs, the bpftrace reference guide and the BCC tools.

Related: Observability in 2026: OpenTelemetry Graduated, the Collector Is Still v0.162, eBPF Instrumentation Is v0.14, and Datadog Bills $1.12B a Quarter, microVM Production Incidents: 4 Postmortems From My Fleet and How to Fix Kubernetes CrashLoopBackOff.