systemd Explained for DevOps (2026): Units, Timers, journald, v262

Nov 10, 2026 · 12 min · Ajay Kumar

Every Linux box I have ever been paged about, from a 2014 EC2 instance to the hosts running my Firecracker fleet today, ended up with me typing systemctl status within the first minute. systemd is the least glamorous part of the infrastructure stack and the one that most often decides whether a service comes back after a crash, whether a log line survives a reboot, and whether a compromised process can write to /etc. This post explains systemd for DevOps engineers as it stands in November 2026: what units and targets are, how service restarts and rate limits actually behave, why timers beat cron for most jobs, how journald stores logs, how cgroup v2 resource limits work, what socket activation buys you, and which hardening directives are worth adding to every unit you write.

What is systemd, and which version are you running?

systemd is the init system (PID 1) and service manager on almost every mainstream distribution, plus a family of helpers: journald for logs, logind for sessions, networkd and resolved for networking, timesyncd, tmpfiles and more. PID 1 reads declarative unit files, builds a dependency graph, starts things in parallel, puts each service in its own cgroup, and supervises it until shutdown.

The current upstream release is v262, published on 22 September 2026, with point releases (262.2 shipped on 5 November) maintained in the separate stable branch. Upstream ships roughly every three months, but distributions freeze a version for years, so the systemd you debug is usually older than the man page you are reading. I checked the archive APIs for Ubuntu and Debian, and Red Hat's RHSA-2026:18153 errata for RHEL 10:

Distribution systemd version Source
Ubuntu 24.04 LTS (noble) 255.4 Launchpad
Ubuntu 26.04 LTS (resolute) 259.5 at release, 259.9 in updates Launchpad
Debian 12 (bookworm) 252.39 sources.debian.org
Debian 13 (trixie) 257.13 sources.debian.org
RHEL 10 257 Red Hat errata
Debian unstable (sid) 262.2 sources.debian.org

Run systemctl --version before trusting any blog post, this one included. Two recent upstream changes matter to anyone running fleets. Release 258 (17 September 2025) removed cgroup v1 entirely, so cgroup v2 is always mounted. Release 260 (17 March 2026) removed System V init script support: the sysv generator, rc-local.service and the systemd-sysv-install hook are gone, and the kernel baseline moved to 5.10. If you still have an /etc/init.d/ script or an /etc/rc.local doing real work, it will silently stop running the day your distribution picks up 260.

systemd releases, June 2024 to September 2026 (red: breaking removals) v256 11 Jun 2024 adds run0 v257 10 Dec 2024 Debian 13, RHEL 10 v258 17 Sep 2025 cgroup v1 removed v259 17 Dec 2025 Ubuntu 26.04 LTS v260 17 Mar 2026 SysV, rc.local removed v261 19 Jun 2026 IMDS client v262 22 Sep 2026 current Distros freeze a version for years: check systemctl --version before trusting any man page.
Upstream systemd releases from the GitHub release pages and the NEWS file; distribution versions from the Launchpad, Debian sources and Red Hat errata listed in the table above.

Units, targets and where unit files live

Everything systemd manages is a unit, named name.type. The types you meet weekly are .service (a process), .socket (a listening socket that can start a service), .timer (a schedule that starts a service), .target (a named group used as a synchronisation point), .mount and .slice (a node in the cgroup tree). Targets replace runlevels: multi-user.target is the old runlevel 3, graphical.target is 5, and network-online.target is the one people get wrong, because After=network.target only orders you after the network stack is configured, not after an address is reachable.

Two kinds of relationship do all the work, and they are independent. Wants= and Requires= say "start that too"; After= and Before= say "in this order". A unit that Wants=postgresql.service without After=postgresql.service starts both at once, which is the source of a lot of first-boot flakiness.

Unit files load from three places, in increasing priority: /usr/lib/systemd/system/ (package-owned, never edit), /run/systemd/system/ (runtime) and /etc/systemd/system/ (yours). To change a packaged unit, do not copy it. Run systemctl edit nginx.service, which writes a drop-in at /etc/systemd/system/nginx.service.d/override.conf that survives package upgrades, and systemctl cat nginx.service shows the merged result. After editing any file by hand, systemctl daemon-reload is mandatory; systemd caches parsed units and will happily keep running the old definition.

How service types and restart policies actually behave

Type= tells systemd when a service counts as started. simple considers it started the moment fork() succeeds, before your binary has even been executed, so a missing binary shows up as a failure after "started". exec waits until execve() succeeds, which is what you usually want for a plain long-running daemon. notify waits for the process to send READY=1 over sd_notify, oneshot waits for the process to exit (good for migrations and setup scripts), and forking exists for legacy daemons that double-fork and write a PID file. v262 tightened notify-reload: such services must now catch or block their reload signal before sending READY=1, or they fail with a protocol error.

Restart=on-failure is the policy most services want. The defaults around it surprise people. RestartSec= defaults to 100ms, and start rate limiting defaults to StartLimitBurst=5 within StartLimitIntervalSec=10s (systemd-system.conf). A service that crashes on startup therefore burns its five attempts in about half a second, and systemd then refuses to start it at all with start-limit-hit until you run systemctl reset-failed. Note that the start limit settings belong in [Unit]. systemd still parses the legacy StartLimitInterval= and StartLimitBurst= spellings under [Service] for compatibility, but StartLimitIntervalSec= placed there is an unknown key and is ignored with a warning, which is how many hand-written rate limits silently do nothing.

The fix is a realistic RestartSec= plus exponential backoff, available since v254 through RestartSteps= and RestartMaxDelaySec=. The man page suggests 3 to 5 steps. v262 also moved rate-limit timers to CLOCK_BOOTTIME, so time spent suspended now counts toward the interval, which mostly matters for laptops and edge devices. I wrote up the full failure workflow, with exit codes such as 203/EXEC and 226/NAMESPACE, in the companion guide on how to fix a systemd service that fails to start.

systemd timers vs cron: which should you use?

Cron still works and is still installed on most servers. Timers win for anything you need to operate rather than merely schedule, for four reasons. Every run is a normal service, so it gets the same logs in the journal, the same cgroup accounting, the same resource limits and the same hardening as a daemon. systemctl list-timers shows the last and next run of every job on the box, which no cron implementation gives you. Persistent=true stores the last trigger time on disk and catches up a missed run after the machine was off (systemd.timer), which is what anacron was for. And RandomizedDelaySec= spreads a fleet's 03:00 jobs so 400 machines do not hit the same S3 bucket in the same second.

Two details trip people up. AccuracySec= defaults to one minute, so a timer can fire up to 60 seconds after its scheduled time to let systemd batch wake-ups; set AccuracySec=1s if you care. And OnCalendar= syntax is not cron syntax: *-*-* 03:00:00 is daily at 03:00, Mon..Fri 09:00 is weekdays. Test expressions with systemd-analyze calendar "Mon..Fri 09:00" --iterations=3 before deploying them.

Here is the pattern I use for a nightly job: a oneshot service with resource limits and hardening, and a timer that triggers it.

# /etc/systemd/system/db-backup.service  (tested against systemd 255 to 262)
[Unit]
Description=Nightly database backup
Wants=network-online.target
After=network-online.target
# Rate limits belong in [Unit], not [Service]
StartLimitIntervalSec=10min
StartLimitBurst=3

[Service]
Type=oneshot
User=backup
Group=backup
EnvironmentFile=/etc/db-backup/env
ExecStart=/usr/local/bin/db-backup --target s3://example-backups/nightly
# cgroup v2 limits: throttle at 1G, kill at 1.5G, half a CPU
MemoryHigh=1G
MemoryMax=1536M
CPUQuota=50%
IOWeight=50
# Hardening: read-only OS, private /tmp, no privilege escalation
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
# Creates /var/lib/db-backup owned by User=, writable despite ProtectSystem
StateDirectory=db-backup
CapabilityBoundingSet=
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
SystemCallFilter=@system-service

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run db-backup nightly

[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=15min
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer, not the service: systemctl enable --now db-backup.timer, then systemctl list-timers db-backup.timer to see the next run.

journald and journalctl: where your logs actually go

journald captures stdout and stderr of every service, kernel messages and syslog calls, and stores them in an indexed binary format with structured fields (_SYSTEMD_UNIT, _PID, PRIORITY, _BOOT_ID). The useful queries are short: journalctl -u nginx.service -b (this boot), -p err (errors and worse), --since "1 hour ago", -f (follow) and -o json when you are piping into something else.

Persistence is the setting to check on every new image. With the default Storage=auto, journald writes to /var/log/journal/ only if that directory exists; otherwise logs live in /run/log/journal/ and vanish on reboot, which you discover on the morning after a kernel panic. Disk use is capped by SystemMaxUse=, which defaults to 10 percent of the filesystem, capped at 4G. On small root volumes that is still a lot; set it explicitly. If you ship logs to Loki or OpenSearch, keep journald as the local buffer and let the shipper read from it, which is how the Fluent Bit setup works on nodes.

cgroups and resource limits: cgroup v2 only since v258

Every unit runs in its own cgroup under a slice: system.slice for services, user.slice for logins, machine.slice for VMs and containers. That is why systemctl stop reliably kills a service's grandchildren, and why systemd-cgtop can show per-service CPU and memory with no agent installed.

Since v258 there is only the unified cgroup v2 hierarchy, and Kubernetes followed: from v1.35 the kubelet's failCgroupV1 defaults to true, so a cgroup v1 node refuses to run the kubelet. The resource directives map directly to cgroup v2 controllers (systemd.resource-control). MemoryHigh= throttles and reclaims; MemoryMax= is the hard wall where the OOM killer fires inside the unit, and the man page recommends MemoryHigh= as the main control. CPUQuota=200% means two full CPUs. TasksMax= caps processes and threads, defaulting to 15 percent of the kernel's PID and thread limits, which is the setting that quietly stops a fork bomb or a leaky thread pool. You can change limits live with systemctl set-property db-backup.service MemoryMax=2G.

This is the same machinery Kubernetes uses for pod limits, so the mental model carries over directly to requests and limits.

What is systemd socket activation, and when is it worth it?

With socket activation, systemd opens the listening socket itself (.socket unit) and starts the matching service on the first connection, handing over the already-bound file descriptor. Connections that arrive during startup or a restart queue in the kernel instead of being refused. That gives you three things: on-demand start for rarely used services, restarts without dropped connections, and the ability to bind privileged ports without the service ever running as root.

# /etc/systemd/system/myapi.socket
[Socket]
ListenStream=443
# Start one service instance for all connections (Accept=no is the default)
Accept=no

[Install]
WantedBy=sockets.target

The catch is that the application must accept a passed file descriptor (via sd_listen_fds() or the LISTEN_FDS environment variable). Go, Rust and Python all have small libraries for it; many off-the-shelf daemons do not. For most web services behind a load balancer, I skip it.

systemd-analyze and the hardening directives worth using

systemd-analyze is the toolbox people forget exists. blame and critical-chain explain slow boots, verify lints a unit file before you ship it, calendar tests timer expressions, and security scores a service's sandboxing. That last one assigns an exposure level from 0.0 to 10.0, where high means little sandboxing. Run systemd-analyze security on a stock server and most third-party services come back near the top of the scale, because nobody wrote any hardening.

The directives that give the most for the least breakage, all from systemd.exec:

  • NoNewPrivileges=yes blocks setuid escalation.
  • ProtectSystem=strict mounts the whole filesystem read-only except /dev, /proc and /sys; pair it with StateDirectory= or ReadWritePaths= for the directories the service really writes.
  • ProtectHome=yes and PrivateTmp=yes hide home directories and give the service its own /tmp.
  • DynamicUser=yes allocates a throwaway UID at start, so you do not need a system user at all.
  • CapabilityBoundingSet= (empty) drops every capability; add back only what is needed, such as CAP_NET_BIND_SERVICE.
  • SystemCallFilter=@system-service applies a sensible seccomp allow-list.

Add them one at a time, restart, and watch the journal. The usual failure is a path the service writes that you forgot, which shows up as exit code 226/NAMESPACE or a permission error. This is the same principle as a Kubernetes security context: least privilege, declared in config, reviewed in Git.

FAQ

Is systemd better than cron for scheduled jobs?

For servers you operate, usually yes. Timers give each run journal logs, resource limits, hardening and a visible last and next run in systemctl list-timers, and Persistent=true catches up missed runs. Cron is still fine for a quick personal job on a box you do not monitor.

Did systemd remove cgroup v1 and SysV init support?

Yes. cgroup v1 support was removed in v258 (September 2025) and System V init script support, including rc.local, in v260 (March 2026). Distributions on older versions, such as Debian 13 and RHEL 10 on 257, can still ship SysV compatibility until they move to v260 or later, though a distribution may already build without it.

Why does my change to a unit file not take effect?

systemd caches parsed unit files. Run systemctl daemon-reload after editing, then restart the service. If you edited a file under /usr/lib/systemd/system/, a package upgrade will overwrite it; use systemctl edit to create a drop-in instead.

Where are systemd logs stored?

In the journal, under /var/log/journal/ if that directory exists, otherwise in volatile memory under /run/log/journal/. Read them with journalctl -u <unit>; add -b for the current boot.

What I would do this quarter

Before your distribution picks up v260, I would grep every image for /etc/init.d/ scripts and a non-empty /etc/rc.local, and convert them to native units, because that removal fails silently. Then I would make persistent journald and an explicit SystemMaxUse= part of the base image, set RestartSec= and backoff on every service we own, and run systemd-analyze security in CI against our own unit files with a threshold, the same way we lint Terraform. None of this is exciting. It is the layer that decides whether the AI-era stack on top of it comes back after a bad night.

Related: How to Troubleshoot Linux Performance in 60 Seconds (2026 Checklist), Structured Logging Best Practices in 2026: JSON, Trace IDs, PII, Cost and SSH Security Best Practices 2026: OpenSSH 10.6, ML-KEM and FIDO2 Keys.

Spot an error or something out of date? Email me and I will fix it and log it on the corrections page.

I'm Ajay Kumar — I build and operate PandaStack, an open-source Firecracker microVM cloud for AI agents, and write about the infrastructure and tooling around it.

Need this kind of infrastructure work? See what I do or email hello@ajayk.sh.


Related