The best description of this project is the one its authors wrote: the README's first line calls Apache Spark "a unified analytics engine for large-scale data processing." That phrasing undersells what the repository actually holds. Behind that sentence sits a SQL parser, a rule-based and cost-aware query optimizer, a whole-stage code generator, a distributed DAG scheduler, a shuffle machinery, a unified memory manager, and a micro-batch streaming engine — all in Scala, all on the JVM. On top of the core, the same README lists Spark SQL for SQL and DataFrames, a pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing. One process model, one scheduler, one memory manager, stretched from a laptop to thousands of nodes.

For anyone building AI systems, Spark is the tool you reach for the moment a dataset stops fitting on the machine that runs your notebook. Feature engineering over billions of rows, tokenizing a raw crawl before a training run, deduplicating text before it lands in a vector store, joining click logs with labels to assemble shuffled training batches — these are exactly the "too big for a single process" jobs that single-node engines like Polars and Python-native ones like Dask attack from other directions. Spark attacks the same problem from the JVM side: the engine keeps planning on a driver process, ships serialized tasks to executors, and treats everything — batch SQL, ML training data prep, streaming windows — as DAGs of stages.

The repository rewards reading it. The path from one line of SQL to tasks running on worker nodes crosses a handful of clearly named subsystems, and every one of them is greppable in the tree. The two diagrams in this post were generated directly from the master branch of apache/spark, and every link in the walkthrough below points at a real file and line number you can open on GitHub.

The overview path: the DataFrame and SQL API builds logical plans, Catalyst analyzes and optimizes them, physical planning produces a code-generated plan, and the DAGScheduler turns lineage into stages and tasks that run on executors — with the micro-batch streaming engine feeding the same core.

Reading the overview from left to right:

• SparkSession, L63 is the entry point every session starts from; its def sql, L538 turns SQL text into a DataFrame.
• Dataset, L125 carries an unresolved logical plan; each select, L1195 or filter just wraps it in another node.
• QueryExecution, L67 is the state machine of one query: analyzed, L238, optimizedPlan, L361, and finally executedPlan, L404.
• Analyzer batches, L674 resolve names, functions, and types against the catalog.
• Optimizer batches, L564 rewrite the resolved plan at fixed point until nothing changes.
• AdaptiveSparkPlanExec, L71 re-plans the physical plan between shuffle stages using runtime statistics.
• SparkContext.runJob, L2858 is where a finished plan becomes a job on the cluster.
• DAGScheduler.runJob, L1886 splits RDD lineage into stages and submitMissingTasks, L3215 ships them out.
• Executor.launchTask, L551 deserializes each task on a worker and runs it against its partition.

The internals view: Catalyst rule batches, physical strategies, code generation, stage splitting into result and shuffle-map tasks, executor-side storage and memory, and the DStream graph that JobGenerator ticks once per batch interval.

Inside the Optimizer

Catalyst is the reason you can write sloppy SQL and still get fast jobs. The Analyzer, L404 first walks your unresolved plan with fixed-point Resolution batches, L674, binding table names through the catalog and inferring types along the way. The Optimizer, L51 then applies its rule batches, L564 repeatedly — predicate pushdown, column pruning, constant folding, join reordering — until the plan stops changing, and a Finish Analysis batch, L186 cleans up the rest. Only then does JoinSelection, L200 decide, per join, between broadcast, shuffle-hash, and sort-merge execution; large joins typically land in SortMergeJoinExec, L39.

The JVM payoff is WholeStageCodegenExec, L744: whole chains of operators get fused into a single generated Java function, produced by doCodeGen, L780, so a filter and an aggregation over the same rows become one tight loop with no virtual-call overhead between them. The boundaries where fusion cannot happen are the ShuffleExchangeExec, L190 nodes — and those are precisely where AdaptiveSparkPlanExec, L71 waits to re-plan once it can count the actual shuffle output.

Scheduling, Shuffle, and Memory

Below the SQL layer everything is RDDs. An RDD, L84 is a partitioned lineage with just enough information to recompute any lost partition. An action calls SparkContext.runJob, L2858, which hands the DAG to the scheduler. handleJobSubmitted, L2619 builds the final stage via createResultStage, L988, walking shuffle dependencies backward; each shuffle boundary produces a map stage created by createShuffleMapStage, L737. Two task types do the work: ShuffleMapTask, L54 writes its partition's output for the next stage, and ResultTask, L55 returns the final answer to the driver.

Placement is the resourceOffers, L573 loop in TaskSchedulerImpl, L83, which matches pending tasks to the cores that executors have offered. On the worker side, Executor.launchTask, L551 runs each task on a thread pool thread. Shuffle files and cached partitions both flow through the BlockManager, L183, while UnifiedMemoryManager, L58 splits the heap between execution and storage pools and lets them borrow from each other — which is why caching a big dataset and running an aggregation rarely deadlock each other's memory.

Streaming the Micro-batch Way

The streaming/ module is the classic DStream engine, and it is a masterclass in reusing a batch system. A DStream, L64 is just a sequence of RDDs, one per interval. StreamingContext.start, L581 activates the scheduler, and a timer in JobGenerator.generateJobs, L251 walks the DStream graph every interval; terminal nodes like ForEachDStream.generateJob, L47 produce an ordinary Spark job, which JobScheduler.submitJobSet, L145 hands to the same DAGScheduler the batch engine uses. The result: your stream processing literally becomes tiny batch jobs on a clock, with fault tolerance borrowed from RDD recomputation.

Worth knowing when you read modern docs: the README names Structured Streaming as the stream-processing story, treating the stream as an unbounded table over the SQL engine, while the DStream engine diagrammed above remains in the tree as the older micro-batch machinery. Same clock idea, different API layer — and the diagrams in this post map the machinery that both of them eventually land on.

From Install to Your First Model

The fastest path, straight from the README: download a prebuilt package from spark.apache.org (no build needed), then start the Scala shell with ./bin/spark-shell. The README's own smoke test is spark.range(1000 * 1000 * 1000).count(), which should print 1,000,000,000 — that single line exercises session creation, planning, stage scheduling, and execution end to end. Python users get the identical flow with ./bin/pyspark and the same one-liner, and it is the natural first cell before pointing a DataFrame at real training data.

Ready-made examples ship in the repo: ./bin/run-example SparkPi runs the Pi estimator locally, and prefixing MASTER=spark://host:7077 points the same command at a standalone cluster (or yarn). If you want to build from source instead, the README specifies ./build/mvn -DskipTests clean package with Apache Maven, and ./dev/run-tests once built. MLlib ships in the same distribution, so once the shell runs, the "first model" is a matter of assembling a DataFrame and letting the engine distribute the rest — no extra cluster software to install for a first run.

Honest Limits

First, the JVM tax. The engine core is Scala and requires a modern JDK to build and run, and the README's build matrix treats Java 21 as the baseline; a source build also couples you to the Hadoop version your cluster runs, because Spark uses the Hadoop core library to talk to HDFS. Second, language drift: the README marks the R API as deprecated, and the connect-based remote sessions show where the project is heading — the classic JVM-embedded session is no longer the only execution mode, but the mature tooling still assumes it. Third, the driver is a real constraint: planning, stage splitting, and task bookkeeping all live in one driver process, so a pathologically wide query plan is a driver-side problem no amount of executors fixes. Fourth, shuffle is materialized — map tasks write files that downstream tasks fetch through the BlockManager, so an unplanned join explosion shows up as disk and network I/O, which is exactly the class of problem AdaptiveSparkPlanExec exists to mitigate at runtime. Finally, micro-batch means latency floors at the batch interval: sub-second streaming is a Structured Streaming and continuous-processing conversation, not a DStream one. None of these limits is a secret — they are all visible in the README or in the source paths cited above, which is exactly why reading the tree pays off before you commit a workload to it.

How It Fits an AI Stack

In a modern ML platform, Spark is the heavy-duty stage between raw storage and anything that trains or serves. It reads from object stores and warehouses, runs the join-and-aggregate grind of feature engineering, and hands off parquet shards or training tables to whatever consumes them. [Kafka](/kafka/) is its natural streaming front door, with the micro-batch engine above picking up where the broker leaves off. For interactive federation over the same lake, [Trino](/trino/) answers the ad-hoc SQL while Spark does the long-running batch; and if your data comfortably fits one machine, the honest comparison points are [Polars](/polars/) and [Dask](/dask/) before you reach for a cluster. The source tree teaches the shape of every large-scale engine that came after it — plan, optimize, schedule, shuffle — which is why it remains worth an afternoon of reading.

Enjoyed this post? Never miss out on future posts by following us