Every Python developer eventually hits the same wall: a table too big for memory, a join that takes twenty minutes, a group-by that spills into swap. The usual answer has been to rent a bigger machine or to retreat into Spark. Polars, the most-starred DataFrame project in the Rust ecosystem, takes a different bet: rewrite the entire engine in Rust with Apache Arrow memory at the bottom, expose a thin Python surface through PyO3, and make deferred execution the default rather than an afterthought. The result is a library where import polars as pl feels like pandas but executes like a modern analytical database.

The repository layout makes the architecture claim explicit. The Python code lives under py-polars/src/polars/ — a few dozen hand-written modules that are mostly argument checking, docstrings, and delegation. Everything that touches actual bytes lives under crates/, a workspace of more than thirty Rust crates: polars-core for the columnar primitives, polars-plan for query representation and optimization, polars-expr for expression evaluation, polars-mem-engine and polars-stream for the two execution engines, polars-io for readers, and polars-python for the PyO3 bridge. The Python layer is deliberately boring; the interesting engineering is all one unsafe boundary away.

This post is a source tour of that boundary. We walk from the DataFrame a user constructs, through the lazy query stack that turns group_by(...).agg(...) into an optimized logical plan, down into the chunked Arrow columns where kernels actually run — and back up through the streaming engine that assembles results as morsel batches. Two diagrams anchor the tour; both were generated from the real repository tree on main (2.x-era, October 2026), and every symbol cited below was verified against the checked-out sources.

Polars from 10,000 feet: the Python surface defers everything into a plan, the planner optimizes it, and the engines execute against Arrow-backed columns.

Reading the overview from left to right:

• DataFrame, L207 — the eager 2-D table users touch first. Almost every method is a shim over a compiled counterpart; the class itself holds no data logic.
• lazy, L10550 — df.lazy() converts eager work into a LazyFrame, L196, the deferred query builder that records operations instead of running them.
• collect, L2075 — the single entry point that materializes a LazyFrame. Its signature alone tells the 2.x story: an engine parameter accepts "auto", "in-memory", "streaming", or "gpu".
• Expr, L161 — column expressions such as pl.col("a").sum(). Built by col, L407, each call crosses into Rust immediately.
• DslPlan, L32 — the language-agnostic query DSL. Python, R, Node, and SQL all lower into this enum, which is why the optimizer is written exactly once.
• IR, L56 — the resolved, arena-allocated logical plan that optimize, L112 rewrites with pushdown and common-subexpression rules.
• DataFrame (Rust), L85 — the real storage: a vector of Series, L151, each wrapping a ChunkedArray, L140 that mirrors Arrow buffers.
• execute_graph, L285 — the streaming engine’s driver, which runs the optimized plan as an asynchronous graph of operators.

The detail view: every arrow above traces a real call path through the two codebases, from `df.group_by()` down to Arrow chunks and back.

Why a Rust Core Changes the Data Model

The single most consequential line in polars-core is the definition of a column. At Series, L151 you find pub struct Series(pub Arc<dyn SeriesTrait>) — a type-erased, reference-counted handle to a trait object. Every concrete dtype implements SeriesTrait, L212, and behind it sits ChunkedArray, L140, which stores data as a sequence of Arrow-compatible chunks rather than one contiguous array.

Two design consequences follow. First, cheap concatenation and parallel-friendly processing: appending a million rows does not copy the existing buffer, it adds a chunk, and kernels iterate chunks across threads. Second, zero-copy interop: because chunks are Arrow, handing a column to numpy, Arrow C Streams, or a training loop backed by PyTorch can be a pointer hand-off rather than a conversion. The logical type system lives in DataType, L90, an enum that spans primitives, temporals with time zones, categoricals, lists, structs, and decimal types — a reminder that a “DataFrame library” is really a small type system wearing a table costume.

The eager API keeps this machinery invisible. The GroupBy, L117 class wraps split-apply-combine, but even eager operations route through the compiled layer, so the Python interpreter rarely loops over data. That is the quiet part of Polars’ speed: it is not just SIMD kernels, it is the absence of a Python frame per row.

The Lazy Plan: Where Queries Become Graphs

The lazy surface is where Polars stops resembling pandas. LazyFrame, L196 records operations; nothing runs until collect, L2075. Inside Rust, the recorded operations form a DslPlan, L32 tree. The function to_alp_impl, L193 (“ALP” here means the arena logical plan) resolves names, expands selectors, checks schemas, and produces the compact IR, L56 representation that everything downstream consumes.

Then the optimizer goes to work. The rule driver at optimize, L112 invokes passes like PredicatePushDown, L25, which slides filters down the tree toward the scan nodes, and ProjectionPushdownVisitor, L128, which discards columns nobody will read. The payoff compounds with the IO layer: a pushed-down filter plus a projected column list lets the Parquet reader skip entire row groups using footer statistics, so a scan over a 100 GB dataset may touch a few hundred megabytes. This is the same core trick databases have used for decades, applied to a DataFrame API — and it is why the lazy API is not merely a syntax variant but the performance-critical path.

Optimization is also where the SQL story lives. The SQLContext, L229 parses SQL into the same DslPlan, so pl.sql() queries benefit from identical rewrites. One planner, three dialects.

From Install to Your First Model

Polars ships prebuilt wheels for Linux, macOS, and Windows, so installation is a single command and there is no Rust toolchain involved:

pip install polars

The canonical first query reads a dataset, filters, aggregates, and materializes — all in one deferred chain:

import polars as pl

q = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("ts") >= pl.datetime(2026, 1, 1))
    .group_by("user_id")
    .agg(
        pl.col("amount").sum().alias("spend"),
        pl.col("event").len().alias("hits"),
    )
    .sort("spend", descending=True)
)

df = q.collect(engine="streaming")

Note the shape of this snippet. scan_parquet — defined at scan_parquet, L520 — opens nothing eagerly; it just plants a scan node in the plan. The engine choice at collect time selects the streaming executor, which keeps memory bounded on datasets far larger than RAM. If you have been feeding model training from Hugging Face datasets, this is the same mental model: declare the transformation, let the engine decide how to stream it. Converting the result to numpy or Arrow for a training loop is one .to_numpy() or __arrow_c_stream__ call away, and for feature-table workflows that previously ran on pandas, the API mapping is close enough that most scripts translate in minutes.

The eager equivalents remain for interactive work: read_csv at read_csv, L74 and read_parquet at read_parquet, L87 return a DataFrame immediately. Under the hood they construct the same scan machinery and collect it, so even “eager” IO gets pushdown for free.

Bindings and the Two Engines

The bridge between the layers is unusually clean. Python imports polars._plr, which — per the swap logic at the bottom of _plr.py, L44 — replaces its own module entry with the compiled _polars_runtime package built from crates/polars-python. After that, plr.col(...), PyDataFrame at PyDataFrame, L20, and PyExpr at PyExpr, L44 are direct PyO3 types. The expression factory in Python literally calls wrap_expr(plr.col(name)) at col.py, L67 — there is no second expression implementation to keep in sync.

At collect time, the engine selector, L87 resolves "auto" into one of two executors. The in-memory engine implements the Executor trait, L10 with one struct per operator — GroupByExec, L23 is a good example — and materializes intermediate results as whole frames. The streaming engine instead lowers the IR through lower_ir, L235 into PhysNodeKind, L171 nodes connected as an asynchronous graph, where data moves between operators as Morsel, L83 batches — small ordered chunks that flow under backpressure, which is how the engine keeps memory flat on out-of-core workloads. Physical expressions for both engines are produced by create_physical_expr, L129, so the kernel layer is shared.

Honest Limits

Polars is not pandas, and pretending otherwise misleads teams mid-migration. The statistical and modeling ecosystem — statsmodels, seaborn, plotting conventions, thousands of small utilities — still expects pandas objects, and while interop is good, it is a conversion, not a shared object. The gpu engine option delegates to an external cudf-polars backend rather than shipping in-tree. The SQL dialect covers a large but not complete subset of what DuckDB or Postgres accept. And the 2.x-era codebase is visibly in motion: the Python package moved from python/polars to py-polars/polars to py-polars/src/polars across major versions, the streaming flag on collect was removed in favor of the engine parameter, and any source citation from an older blog post is likely stale today. The entry-point swap in _plr.py and the split runtime packages (_polars_runtime_64, _polars_runtime_32, a compat build) are recent architecture, not settled history.

There are also the inherent constraints of the design. Row-wise Python UDFs remain the escape hatch that kills performance, because every call crosses the boundary the rest of the system is built to avoid. String-heavy workloads are dramatically better than the GIL-bound era but still pay for variable-width data. And the deferred model asks something of users: code written eagerly by default will not get pushdown benefits, so the discipline of starting from scan_* matters.

None of that changes the core verdict. For tabular work — ETL, feature engineering, dataset preparation, aggregations that used to justify a Spark cluster — Polars is the fastest path between intent and result that Python currently has, and the source shows exactly why: a type-erased Arrow core, one optimizer shared by every front end, and an execution layer that treats memory as the scarcest resource in the room.

Enjoyed this post? Never miss out on future posts by following us