Most machine learning stacks never admit how much time they spend converting data. NumPy wants one layout, pandas wants another, your trainer wants tensors, and the Parquet file on disk wants a third. Every conversion is a memcpy nobody budgeted for. Apache Arrow attacks exactly that problem: it is a specification for a columnar, in-memory representation — and a working implementation of it — so that a column stored on disk, shipped over a network socket, or handed to a GPU-adjacent library looks like the same bytes, laid out the same way, with the same null encoding. When the formats agree, “reading a file” can degenerate into pointing at memory you already have.

What makes Arrow worth a source tour is that the spec and the code are unusually close together. The C++ implementation in cpp/src/arrow/ is the reference behavior: buffers, types, validation, IPC framing, a compute kernel library, and a dataset scanner that hides files, partitions, and row groups behind one API. The Python package pyarrow is not a reimplementation — it is a thin Cython shell over the C++ objects, which is why a pyarrow.Table can be handed to pandas, Polars, DuckDB, or even the Hugging Face datasets library without anyone copying columns first. In this post we walk the repository from the Python surface down to the buffer pool, with two generated architecture diagrams to anchor the tour.

The two diagrams below were generated from the actual file tree of the repository. The overview shows the six moving parts a data engineer touches; the detail view zooms into the dataset layer, the CSV stack, and the memory infrastructure that make the zero-copy story real.

Apache Arrow end to end: pyarrow's Cython surface wraps the C++ core, whose arrays and record batches serialize through flatbuffer-based IPC, while the dataset layer drives format readers and compute kernels.

Reading the overview from left to right:

• Python tabular surface. Everything starts in table.pxi, where the Cython classes live. The Table class, L4064 inherits from _Tabular, and constructors like Table.from_arrays, L4762 wrap C++ arrays behind a friendly signature. Relational sugar like Table.join, L5644 and Table.group_by, L5603 actually delegates to the compute layer rather than reinventing joins in Python.
• Array wrapper. In array.pxi, the Array class, L1122 exposes the honest mechanics: Array.buffers, L1955 hands back the raw validity and data buffers, Array.to_numpy, L1782 decides whether a conversion is zero-copy or not, and Array.validate, L1920 checks structural invariants. If you have ever wondered what a “column” physically is, this is the file to read.
• C++ Table core. On the C++ side, table.cc implements chunked composition. Table::CombineChunks, L599 consolidates fragmented columns, ConcatenateTables, L407 glues tables vertically, and TableBatchReader::ReadNext, L687 streams a Table back out as record batches — the unit that everything downstream actually consumes.
• Record batches. record_batch.cc defines the column-aligned row group with a shared row count. RecordBatch::FromStructArray, L250 shows how a struct array of N fields becomes an N-column batch, and RecordBatch::SelectColumns, L423 is a pure metadata operation — no bytes move, because each column is just a pointer to a buffer.
• The wire. Serialization lives in writer.cc. The RecordBatchSerializer class, L131 walks every buffer, writes its flatbuffer metadata, and pads body segments to 8-byte alignment; WriteRecordBatchStream, L874 sequences those messages into the stream format. This framing is what “Arrow IPC”, Feather V2, and a lot of network-facing data planes share.
• The read side. In reader.cc, the ArrayLoader class, L173 replays that walk in reverse, attaching buffer views to freshly typed arrays, while StreamDecoderInternal, L1001 handles the incremental case for sockets and chunked streams. Read a file fast enough and you never materialize more than the output you asked for.
• Dataset layer. dataset.cc turns “scan this directory” into an object model: Dataset::NewScan, L204 returns a builder, and fragments implement physical reads — see InMemoryFragment::ScanBatchesAsync, L106 for the trivial case. The async scanner in scanner.cc pushes fragments through a fan-out so batching stays ahead of the consumer; AsyncScanner::ToTable, L402 is the convenience exit point.
• Format readers. Each file format plugs in as a FileFormat. The parquet one, file_parquet.cc, exposes ParquetFileFormat::ScanBatchesAsync, L592, which is where row-group pruning and column projection get applied. The CSV stack is pure Arrow: the streaming reader in csv/reader.cc (constructed via StreamingReader::Make, L1230) parses byte blocks, decodes columns, and emits batches incrementally.
• Memory pools. Under everything sits memory_pool.cc, where SystemMemoryPool, L536 and the mimalloc-backed alternative at MimallocAllocator, L391 account for every allocation the library makes. Arrow tracks bytes deliberately: zero-copy only saves time if you can see where the copies are.

The detail view: kernels, fragments, and the buffer economy

The detail graph adds the array builders, IPC message envelope and dictionary memo, Feather compatibility, the function dispatch path, dataset discovery and writers, the CSV parser-to-decoder stack, and the memory pools underneath.

Two things deserve a closer look than the overview can give. First, the compute subsystem. Arrow’s kernels are typed functions with a dispatch table: compute/function.cc resolves an argument list to the best kernel via Function::DispatchBest, L311, compute/kernel.cc initializes per-call state in Kernel::InitAll, L62 and resolves output types in OutputType::Resolve, L447, and compute/exec.cc packages heterogeneous values into an ExecBatch::Make, L215 for iteration by ExecSpanIterator::Init, L344. This is why a filter or projection can be pushed down into the dataset scan instead of running afterwards in Python: the scanner holds an expression, and expression evaluation is just kernel dispatch.

Second, the dataset factory pattern. Discovery in discovery.cc — DatasetFactory::Inspect, L58 peeks at sources to infer a unified schema, then DatasetFactory::Finish, L68 materializes a dataset. Partitioning schemes (directory-based or Hive-style) fold into that schema, which is how pyarrow.dataset gives you partition-pruned scans of a tree of files without you writing a manifest. The writer side, dataset_writer.cc, handles the boring-but-critical constraints: max rows per file, max files open at once, and naming.

From Install to Your First Model

Arrow installs like any wheel: pip install pyarrow. The first session usually looks like this, straight from the objects we just toured:

import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq

# Build a table from Python objects (table.pxi -> table.cc)
table = pa.table({"model": pa.array(["bert", "gpt2", "llama"]),
                  "params": pa.array([110, 124, 6700])})

# Stream it through IPC: writer.cc frames, reader.cc rebuilds
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, table.schema) as writer:
    writer.write_table(table)
buf = sink.getvalue()
rt = pa.ipc.open_stream(buf).read_all()
assert rt.equals(table)          # schema and buffers agree

# Scan a partitioned Parquet tree with pushdown (dataset layer)
dataset = ds.dataset("runs/", format="parquet",
                     partitioning="hive")
small = dataset.to_table(
    filter=ds.field("params") > 500,
    columns=["model"])

# Exit to pandas only at the boundary
df = small.to_pandas()

Three things happen under the hood worth noticing. The IPC round-trip above serializes through flatbuffers and rebuilds arrays via the loader we cited — the buffer count and sizes come back identical, because the format is deterministic. The ds.dataset(...) call goes through the factory path we quoted: inspect, infer schema, finish. And the filter compiles to an Expression that the scanner evaluates per batch with compute kernels, so partition and row-group pruning happen before your Python code sees a single row. For a training-adjacent example, Hugging Face’s datasets library is Arrow underneath — its memory-mapped shards are Arrow IPC files, which is the same machinery in different clothes.

A practical pattern for inference engineering: land your prompts, token tables, or embedding caches as partitioned Parquet, then stream them into the evaluator with dataset.to_batches(). Because batches are the native unit, you can feed a C++ or CUDA consumer directly through the C data interface without ever involving Python objects.

Where Arrow sits in the ecosystem

Arrow’s columnar layout has quietly become the interchange standard of the Python data stack. Polars executes queries on Arrow arrays natively, and DuckDB can consume them with no serialization at all — query results materialize as Arrow tables across a zero-copy bridge. The Hugging Face datasets library owes its memory-mapped multi-terabyte caching to Arrow IPC files on disk. And pandas interoperates through to_pandas() / from_pandas(), with the newer pandas Arrow-backed dtypes cutting conversion cost further. If your stack includes numpy for tensors, the clean boundary is: Arrow for columns at rest and in transit, NumPy for math inside your own code. Feather files you may have used are Arrow IPC files with a compatibility shim — ipc/feather.cc keeps the old API name but the V2 framing is the wire format we toured.

Honest Limits

Arrow is a format and a toolbox, not a database. There is no storage engine, no transactions, no index that answers point lookups quickly — a column scan means scanning the column. The compute kernel library covers relational basics (filters, projections, aggregations, joins, hash operations) but it is not a full query planner; complex analytical queries belong in DuckDB or Polars, ideally consuming Arrow arrays so nothing is copied. The dataset layer’s strength is also its ceiling: it scans files with pushdown filters, but anything requiring cross-file statistics or adaptive ordering still needs a real engine. Python’s pyarrow is Cython over C++, so the zero-copy claims are real, but the moment you call to_pylist() or to_pandas() on object-dtype columns, copies happen — check to_numpy(zero_copy_only=True) when you want the contract enforced. IPC streams are not random-access: use the file format if you need to seek. Finally, the C data interface is stable, but fast-moving corners — the dataset API is explicitly marked unstable in its docstring, and extension types have compatibility caveats across languages — deserve a test before you commit to them in production.

Enjoyed this post? Never miss out on future posts by following us