Airflow vs Kubeflow vs Prefect vs Dagster (2026): 6 Orchestrators Compared

Every few months someone on a team I am helping asks whether they should "move off Airflow", and the honest answer is usually a question back: off it to what, and to fix which problem? This post compares the six ML and data pipeline orchestrators that keep coming up in 2026, Airflow 3, Kubeflow Pipelines, Prefect, Dagster, Flyte and Metaflow, using the versions that actually shipped this autumn. I will cover the one design choice that separates them (tasks versus assets), how each one sits on Kubernetes, what the managed versions cost, where LLM and agent workflows fit, and which tool I would hand to which kind of team.

Which pipeline orchestrators matter in 2026?

The short list has not changed much in three years. What changed is that every project shipped a major rewrite, so blog posts from 2023 describe software that no longer exists.

Airflow 3.0 landed on 22 April 2025 and 3.4.0 shipped on 6 November 2026. Airflow 2 reached end of life on 22 April 2026 according to the support table in the Airflow README, so anyone still on 2.x is now running unsupported software. Dagster has been on the 1.x line for years and is at 1.13.29 after 1.13.0 in April. Prefect 3.0 shipped in September 2024 and is at 3.9.2. Flyte rebuilt itself as Flyte 2 (the v2.0.0 IDL release was 22 December 2025), Kubeflow Pipelines reached 2.18.0 on 28 October 2026, and Metaflow keeps shipping small 2.19.x releases.

Tool	Latest release (date)	Core abstraction	Runs where	Cheapest managed option	Source
Apache Airflow	3.4.0 (6 Nov 2026)	Dags of tasks, plus assets	Anywhere; KubernetesExecutor or Celery	MWAA small env $0.49/hour	release, MWAA pricing
Kubeflow Pipelines	2.18.0 (28 Oct 2026)	Containerised components in a DAG	Kubernetes only	None from the project	release
Prefect	3.9.2 (6 Nov 2026)	Python flows and tasks, optional assets	Workers on any infra	Hobby free, Starter $100/month	releases, pricing
Dagster	1.13.29 (5 Nov 2026)	Software-defined assets	Anywhere; Helm chart for K8s	Solo $10/month plus $0.040/credit	releases, pricing
Flyte 2	SDK 2.13.1 (9 Nov 2026)	Pure Python async tasks	Union.ai today; OSS backend "coming soon"	Union.ai	flyte-sdk, README
Metaflow	2.19.41 (6 Nov 2026)	Python flows of steps	Deploys to Argo, Step Functions, Airflow, Kubeflow	Outerbounds	releases, docs

All six are Apache-2.0. None of them is a dead end. The differences are in what they make easy.

Tasks vs assets: the design choice that matters most

A task-centric orchestrator asks "what should run, and in what order?" An asset-centric one asks "what data should exist, and is it fresh?" That sounds philosophical until you are paged at 3am and need to know which tables a failed job left stale.

Dagster built itself around the second question. You declare assets (a table, a model file, a feature set) and the functions that produce them, and the scheduler works out runs from the graph. Version 1.13 pushed further with partitioned asset checks and preview "virtual assets" for things like database views that never need materialising. If your pipeline is mostly dbt plus Python plus a warehouse, this model maps onto how you already think.

Airflow started as pure task scheduling and has been bolting on data awareness since datasets arrived in 2.4. In Airflow 3 they are called assets and there is an @asset decorator that creates the asset, a Dag and a task in one go. Version 3.4 also changed a default that will surprise people: single-asset schedules now create one Dag run per event instead of batching all queued events into one run. If you relied on batching, set batch_asset_events=True before upgrading.

Prefect sits in between. Flows and tasks are plain Python functions, and since 3.x there is an optional @materialize decorator in prefect.assets that records lineage across flows. Kubeflow Pipelines, Flyte and Metaflow are task (or step) graphs where artifacts are typed outputs passed between steps, which is closer to how an ML engineer thinks about a training run than how a data engineer thinks about a warehouse.

Here is the same two-step pipeline in Airflow 3 and Dagster, so you can see the difference rather than read about it:

# Airflow 3.4.0 (apache-airflow==3.4.0): asset-first authoring with the Task SDK
from airflow.sdk import asset

@asset(uri="s3://ml-data/raw/events.parquet", schedule="@daily")
def raw_events():
    ...  # pull yesterday's events and write the parquet file

@asset(uri="s3://ml-data/features/user_features.parquet", schedule=[raw_events])
def user_features():
    ...  # runs once per raw_events update (one run per event by default in 3.4)


# Dagster 1.13.29 (dagster==1.13.29): the same graph as software-defined assets
import dagster as dg

@dg.asset
def raw_events_dg() -> None:
    ...  # same work; Dagster tracks freshness and lineage per asset

@dg.asset(deps=[raw_events_dg])
def user_features_dg() -> None:
    ...  # downstream; materialise on a schedule or with declarative automation

defs = dg.Definitions(assets=[raw_events_dg, user_features_dg])

They look similar on screen. The difference shows up in the UI and in operations: Dagster's home page is the asset graph with freshness, while Airflow's is still a list of Dags and runs.

What changed in Airflow 3 (and why it matters for upgrades)

Airflow 3 is the biggest release the project has had. The webserver became an API server started with airflow api-server, tasks can no longer open sessions on the metadata database directly, SubDAGs and SLAs are gone (replaced by task groups, assets and Deadline Alerts), and catchup now defaults to False. The official upgrade guide says to get to Airflow 2.7 or later first and to run the AIR301 and AIR302 ruff rules over your Dags.

The 3.x minors have been substantive rather than cosmetic. Airflow 3.1 added a human-in-the-loop operator suite (HITLOperator, ApprovalOperator), which matters if you are wiring LLM steps that need a person to sign off. Airflow 3.4 adds Task Loops (AIP-111) for do-while style repetition, an experimental Dag Importer so you can author Dags in TypeScript, Go, Java or YAML, and makes psycopg3 the default Postgres driver with statement preparation disabled so it works behind transaction-mode PgBouncer. It also moves new XCom rows into an xcom_v2 table, which makes downgrades slow. Upgrade forward only, with a database backup.

If you are running Airflow on Kubernetes, my Airflow 3 Helm chart guide walks through chart 1.22.0, executor choice, git-sync and the 2.x to 3.x upgrade step by step.

Is Kubeflow Pipelines worth it if you are already on Kubernetes?

Kubeflow Pipelines is the only tool here that is Kubernetes-only by design. Every step is a container, the DSL compiles to a pipeline spec, and the backend runs it on the cluster. That is either exactly what you want or a lot of ceremony.

Release 2.18 is a good one: artifact previews in the run view, Kubernetes Dynamic Resource Allocation for components (useful for GPU claims, covered in my GPUs on Kubernetes explainer), Pydantic inputs and outputs, and export and import of runs between installations. The same release notes are unusually candid about limits. They say PostgreSQL is "not recommended for production" because of an ML Metadata concurrency issue, and that artifact ownership is not fully enforced, so shared storage credentials can expose other tenants' artifacts. If you run multi-tenant KFP, give each namespace its own bucket credentials.

My view: KFP makes sense when the platform team already runs Kubeflow (notebooks, Katib, KServe) and data scientists want a Python SDK that produces containers. It makes less sense as a general data orchestrator. Nobody wants to write a container component to move a CSV.

Prefect, Flyte and Metaflow: the Python-first options

These three share a pitch: write normal Python, decorate it, and let the platform handle retries, caching and remote execution.

Prefect is the lightest to adopt. A flow is a function, a worker pulls work from a pool and runs it on Kubernetes, ECS, Cloud Run or a laptop, and there is no Dag file parsing loop to tune. The Prefect Cloud pricing page lists a free Hobby tier (2 users, 5 deployments), Starter at $100 a month and Team at $100 per user per month. Self-hosting the open-source server is also normal.

Flyte 2 is the most interesting and the hardest to recommend today. The SDK is pure Python with async tasks and flyte.TaskEnvironment, and it is well suited to dynamic fan-out such as agent evaluation sweeps. But the Flyte README says the open-source backend for Flyte 2 is "coming soon" and points to Union.ai for production use. If you need self-hosted, you are on Flyte 1 (v1.16.9 was released in September 2026) or waiting.

Metaflow is the one I would give to a data science team that does not want to learn an orchestrator at all. You write a flow locally, then deploy it to Argo Workflows, AWS Step Functions, Airflow or Kubeflow. It is a developer experience layer over someone else's scheduler, which is a feature.

Orchestrating LLM and agent workflows

Every orchestrator now has an "AI" page. Strip the marketing and the real requirements of LLM pipelines are boring ones: retries with backoff for 429s from the model provider, caching so you do not pay twice for the same prompt, fan-out over thousands of documents or eval cases, and a human approval gate before anything writes to production.

Airflow covers the approval gate natively since 3.1 and handles fan-out with dynamic task mapping, but its scheduler loop adds latency that is fine for nightly batch and wrong for anything interactive. Prefect and Flyte 2 handle dynamic, data-dependent fan-out more naturally because the graph is built at runtime from Python control flow. Dagster is strong when the LLM step produces a durable thing (an embedding index, a fine-tuned model, an eval report) that you want to track as an asset with freshness rules.

What none of them is: an agent runtime. An orchestrator is the right place for a batch embedding refresh or a nightly eval run, as in my piece on evals as the new unit tests. It is the wrong place for a chat agent that needs sub-second tool calls. Keep the long-running, interactive part in a service and let the orchestrator kick off and check the batch parts. This is where platform discipline still pays: retries, idempotency, secrets handling and resource limits matter more when the step in the middle is a non-deterministic model call that costs money each time.

Which orchestrator should your team pick?

Here is how I would decide, with the reasoning rather than a scorecard.

• You already run Airflow 2. Upgrade to Airflow 3. Moving to anything else means rewriting every Dag anyway, and 3.x fixed most of what people complained about (the UI, Dag versioning, tasks talking straight to the metadata database). Airflow 2 is past end of life, so standing still is not an option.
• Analytics engineering team, dbt-heavy, warehouse-centric. Dagster. The asset graph and freshness checks pay for themselves the first time a stakeholder asks why a dashboard is stale.
• Small team, Python services, wants something running this week. Prefect, self-hosted or on Cloud Starter. Lowest ceremony of the group.
• Platform team already operating Kubeflow for ML. Kubeflow Pipelines, with eyes open about the MLMD and multi-tenant caveats in 2.18.
• Data scientists who do not want to learn infrastructure. Metaflow on top of whatever scheduler the platform team already runs.
• Heavy dynamic ML workloads and happy with a vendor backend. Flyte 2 on Union.ai. Revisit self-hosting when the open-source backend ships.

For cost, the managed options are cheap compared with the engineers who would otherwise babysit them. An MWAA small environment at $0.49 an hour is about $358 a month before extra workers (my arithmetic: 0.49 times 730 hours), and Dagster+ and Prefect Cloud entry plans start at $10 and $100 a month respectively. Self-hosting is cheaper only if someone on the team already knows how to run Postgres, PgBouncer and a Helm release well. If that is you, the Kubernetes Jobs and CronJobs explainer covers when you need no orchestrator at all.

FAQ

Is Airflow still relevant in 2026?

Yes. Airflow 3.4.0 shipped on 6 November 2026, the project has by far the largest community of the six, and every major cloud sells a managed version. Its weak spots are latency for interactive work and an asset model that is newer and thinner than Dagster's.

Is Dagster better than Airflow?

For asset-centric data platforms built around dbt and a warehouse, Dagster's model is usually a better fit. For broad task orchestration across many systems, with a large hiring pool and managed offerings on every cloud, Airflow is the safer default. Neither is better in general.

Should I use Kubeflow Pipelines or Airflow for ML?

Use Kubeflow Pipelines if your ML platform is already Kubeflow on Kubernetes and every step is a container. Use Airflow (often with KubernetesPodOperator for the heavy steps) if your ML pipelines are tangled up with data engineering jobs that already live in Airflow.

Can these tools orchestrate AI agents?

They can orchestrate batch work around agents: eval runs, embedding refreshes, fine-tuning jobs and approval steps. They are not designed to host interactive agents that need low-latency tool calls, so keep those in a service and trigger the batch parts from the orchestrator.

This quarter I would get every remaining Airflow 2 install onto 3.4 (or 3.3.2 if you want a release with a patch behind it), pin the Helm chart and image tags, and add batch_asset_events checks to the upgrade checklist. For new greenfield work on a data platform I would prototype in Dagster for a week before defaulting to anything, and I would not start a new self-hosted Flyte 2 project until its backend is actually open source.

Related: MLOps Platforms Compared 2026: MLflow vs Kubeflow vs SageMaker vs Vertex, Kubernetes Jobs and CronJobs Explained and How to Deploy Airflow 3 on Kubernetes with Helm.