Apache Airflow is the open-source orchestrator that turned “cron jobs with dependencies” into a programmable, observable discipline. If you have ever waited on a nightly data load, chained an extraction step to a training job, or retried a flaky API sync until it finally went green, you have almost certainly been leaning on Airflow’s core loop. The project is now deep into its third major era: Airflow 3 split the codebase into a core (airflow-core), a Task SDK (task-sdk), and dozens of provider distributions, and swapped the old Flask UI for a React frontend backed by FastAPI.

That refactoring makes the current main branch a fascinating place to read code. The DAG class you import in your DAG file no longer lives where it did in Airflow 2 — it moved into the Task SDK so task processes can import DAG definitions without dragging the whole database layer along. The scheduler no longer forks workers for every task; instead it hands compact “workloads” to pluggable executors, and each task runs in a supervised subprocess that talks back to a versioned execution API. None of this is hidden, but it is spread across two packages and several long files, which is exactly why a map helps.

This post is that map. I rendered two call-graph diagrams straight from the main tree of apache/airflow (verified at commit time, Airflow 3.4.0dev), then walked every symbol the diagrams reference with grep so the line numbers below point at real code. The first diagram is the 10,000-foot view: who parses, who schedules, who executes. The second zooms into the database models, job runners, and runtime pieces that make one task go from “queued” to “success”.

Overview of Airflow 3: your DAG file is parsed by DagBag, serialized into the database, picked up by SchedulerJobRunner, dispatched through an executor, and executed by the Task SDK runtime — while the FastAPI app serves both the UI and the task execution API.

Reading the overview from left to right:

• DAG authoring starts in the Task SDK: the DAG class, L330 is an attrs-based container of tasks, and the @task collection, L67 turns plain Python functions into operators. Every operator descends from BaseOperator, L670, whose execute(), L1663 is the method your task actually runs.
• When things run is decided by timetables. BaseTimetable, L26 defines the contract, and CronDataIntervalTimetable, L30 is the concrete cron-based schedule most DAGs use.
• Parsing is a separate process: DagFileProcessorManager, L198 supervises a pool of parser subprocesses via its _run_parsing_loop, L560, and DagBag.process_file, L289 does the actual import of a Python file.
• Serialization bridges parsing and scheduling: SerializedDagModel, L285 stores each DAG as JSON — write_dag, L608 — so the scheduler and web UI never re-import your DAG file.
• Scheduling is SchedulerJobRunner, L337: its _run_scheduler_loop, L1981 wakes up, runs _do_scheduling, L2183, creates DagRun, L308 rows, and queues task instances toward executors.
• Execution begins at BaseExecutor, L201, which accepts workloads through queue_workload, L442 and reports finished runs through get_event_buffer, L682.
• The task process is the Task SDK runtime: run(), L1780 executes one task instance end to end, spawned and watched by supervise_task, L3103.
• Everything user-facing exits through the FastAPI app built by create_app, L136, which mounts the REST API, the task execution API, and the React UI backend.

Why a DAG Is the Unit of Work

Airflow’s central abstraction is refreshingly small. A DAG is “a collection of tasks with directional dependencies” — that phrase is the actual docstring of the class at line 330 of the Task SDK’s dag.py. Tasks are added to a DAG either imperatively (task_a >> task_b) or by the decorator machinery; the chain(), L1799 helper is just sugar over the shift operators. Dependencies between tasks are not implicit in time — they are edges the scheduler must respect, and the SDK enforces that a task_id can only be added once per DAG.

What makes the abstraction powerful is what hangs off it. A DAG’s schedule parameter accepts a cron string, a timetable object, or a list of Assets; the Asset class, L267 turns data availability into a first-class trigger, with | and & combinators for “any” and “all” semantics. Data outputs travel between tasks as XCom handles: the XComArg, L50 proxy in the SDK resolves lazily at run time, so you can write train_model(load_data()) and the framework wires the dependency for you. If you have used Dask or Ray for parallel compute, the mental shift is that Airflow does not move your data or run your math — it only decides when each piece runs and what happens when it fails. Tools like Dask and Ray remain excellent executors of the actual computation; Airflow sits above them, which is exactly why provider packages can wrap both.

Parsing, Serialization, and the Scheduler Loop

The parsing layer solves a subtle problem: Python code is unsafe to import inside a long-running orchestrator. A DAG file with a syntax error, an infinite loop, or a heavy import must not take down the scheduler. Airflow’s answer is process isolation. DagFileProcessorManager runs a bounded pool of subprocesses, each of which imports one DAG file via DagBag.process_file and returns a parsing result over IPC. Import errors are captured and surfaced in the UI instead of crashing anything.

Parsed DAGs are then flattened into JSON by SerializedDagModel.write_dag. This is one of the most consequential design choices in the codebase: the scheduler, the API, and the task runtime all read the serialized form, never your original file. It gives the system a stable schema, makes the web UI fast (no re-importing on page load), and lets task processes reconstruct their operator without importing the DAG bundle.

The scheduler itself is a loop over database state. SchedulerJobRunner._do_scheduling consults DagModel rows (the per-DAG bookkeeping record at L316) to decide which DAGs need a new DagRun, then for each run checks which TaskInstance rows (the per-task state machine at L1067) are ready: dependencies satisfied, pool slots free, concurrency limits respected. The update_state, L1353 method on DagRun recomputes whether the whole run is running, success, or failed, while handle_failure, L2786 implements retry bookkeeping with exponential-ish delays up to max_tries. A separate TriggererJobRunner, L214 runs async “trigger” coroutines so sensors can wait on external events without occupying a worker slot — the Trigger model it serves lives at L74.

From Install to Your First Model

The README is explicit that a bare pip install apache-airflow “will not work from time to time” because Airflow is “a bit of both a library and application” — its dependencies are deliberately open, so you must pin them with a constraints file. The documented install is:

pip install 'apache-airflow==3.3.2' \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.10.txt"

The fastest way to a running system is airflow standalone, implemented by StandaloneCommand, L51. It boots the scheduler, the api-server (the Airflow 3 name for the webserver, now a FastAPI + React app), and the triggerer in one process tree, initializes the SQLite metadata database, and configures the SimpleAuthManager. Because that auth manager generates credentials rather than asking you to invent them, the command writes the admin password to a file and tells you where — see find_user_info at L227. If you prefer the manual path, airflow db migrate (the migratedb handler is at L218) applies schema migrations, and user management is delegated to whichever auth manager you configured rather than a built-in users create command.

Now the fun part: a first DAG that actually produces a model artifact. Save this as tiny_model_dag.py in your dags folder:

import json
from pathlib import Path
from datetime import datetime

from airflow.sdk import DAG, task

ARTIFACT = Path("/tmp/tiny_model.json")

with DAG(
    dag_id="tiny_model",
    schedule="@daily",
    start_date=datetime(2026, 10, 1),
    catchup=False,
) as dag:

    @task
    def load_numbers() -> list[float]:
        return [1.0, 2.0, 3.0, 4.0, 5.0]

    @task
    def fit_baseline(xs: list[float]) -> dict:
        mean = sum(xs) / len(xs)
        return {"kind": "mean_baseline", "center": mean, "n": len(xs)}

    @task
    def publish(model: dict) -> str:
        ARTIFACT.write_text(json.dumps(model))
        return str(ARTIFACT)

    publish(fit_baseline(load_numbers()))

load_numbers >> fit_baseline >> publish is expressed by nesting the calls: each decorated function returns an XComArg, and passing it as an argument both defines the edge and moves the value. To run it without the whole orchestration stack, airflow dags test tiny_model 2026-10-29 executes the entire DAG locally for one data interval — the handler is dag_test, L835. Once you are happy, the scheduler takes over and creates a run per daily interval. From here the natural next step is pointing the tasks at real work — a training script that logs to MLflow, for instance — while Airflow handles the when and what-if.

Executors and the Task SDK Runtime

Executors are where scheduling meets the operating system. BaseExecutor defines the contract: accept queued work, run it somewhere, heartbeat, and surface completion events. The bundled LocalExecutor, L120 spawns worker processes per task and drains results in its sync(), L260; the Celery and Kubernetes executors ship as providers with the same interface. Which executor you get is decided once at startup by ExecutorLoader, L57 — its load_executor resolves names like LocalExecutor or celery against installed providers.

The interesting part is what the executor actually receives. In Airflow 3 it is not a fork request; it is a serialized workload object. The executor’s job is to end up running supervise_task in some process: the supervisor spawns a fresh Python child, loads the Task SDK runtime, and the child’s run() does the real work — render templates, call execute() on the operator, push XComs, and transition the task’s state. State writes flow through the execution API (part of the same FastAPI server the UI uses), so a task process never touches the metadata database directly. This is why the Task SDK exists as a separate package: a worker machine needs airflow.sdk and a way to reach the API, not the full core installation.

The FastAPI Era of the UI and REST API

If you knew Airflow 2’s Flask-AppBuilder UI, the new frontend is a genuine architectural break. The server-side entry point is create_app, which assembles middleware, auth-manager wiring, CORS policy, and then mounts routers defined in airflow.api_fastapi.core_api — init_views, L162 registers both an authenticated ui_router (consumed by the React SPA) and a versioned public REST API under /api/v2. The old /api/v1 endpoints are gone entirely; the app returns a hard 404 telling you to migrate. Auth is pluggable via auth managers, and the simple auth manager that airflow standalone uses stores its generated credentials in a file on disk.

Honest Limits

A few things worth knowing before you bet on this codebase, all verifiable in the docs and source:

• Installation needs constraints. The README warns that without a constraints file a plain pip install apache-airflow “will not work from time to time or will produce unusable Airflow installation”. Poetry and pip-tools are explicitly “not currently supported” for installing it.
• The main branch is a dev tree. The tree toured here declares version 3.4.0 in airflow-core/pyproject.toml while the README’s stable install example pins 3.3.2; APIs on main move quickly and line numbers in this post will drift as development continues.
• Breaking changes are real. Airflow 3 removed /api/v1, moved the DAG class and operator bases into the Task SDK, and deprecated importing DagBag from airflow.models.dagbag in favor of airflow.dag_processing.dagbag — an actual deprecation shim in that module.
• It is an orchestrator, not a data engine. Nothing in the core executes your computation beyond one execute() call per task; scale comes from executors and providers, not from the core. The core is also strict about what runs where: task processes deliberately cannot reach the metadata database, which occasionally surprises people debugging connectivity in locked-down clusters.

The detail view: Task SDK authoring types on the left, the parsing and serialization path, the DB models the scheduler lives on, executors spawning supervised task subprocesses, and the FastAPI/CLI surface on the right.

For a different take on structuring LLM-heavy applications, see DSPy, which compiles declarative prompts rather than scheduling tasks; for typed configuration shared between DAG files, Pydantic pairs naturally with Airflow 3, whose Task SDK API datamodels are themselves Pydantic-generated.

Enjoyed this post? Never miss out on future posts by following us