Introduction

Data engineering has become one of the most in-demand and foundational roles in modern technology. Every analytics dashboard, machine learning model, and AI application depends on reliable pipelines that move, transform, and store data at scale. Breaking into the field, however, can feel overwhelming because the tooling landscape is vast, the concepts span distributed systems and databases, and there is no single canonical curriculum. The Data Engineering Handbook was created to solve exactly that problem. It is an open-source repository maintained by DataExpert.io that consolidates the resources a learner needs to go from absolute beginner to confident practitioner.

This guide walks through the repository section by section. We will explore the curated list of more than 25 books, the free beginner and intermediate bootcamps, the projects you can build to demonstrate real experience, the interview preparation materials, the companies and tools that define the modern data stack, and the communities where you can find mentors and peers. Whether you are switching careers, leveling up from analytics, or hiring data engineers and want a shared vocabulary, this handbook is a reliable map of the territory.

What is the Data Engineering Handbook

The Data Engineering Handbook is a community-maintained, open-source resource hosted on GitHub under the DataExpert-io organization. Its stated mission is simple: to give you every resource you need to become an amazing data engineer. The repository is organized as a set of focused Markdown files, each acting as a curated index for one pillar of the discipline. There is no framework to install and no proprietary platform required. Instead, the handbook points you toward high-quality books, free bootcamps, hands-on projects, interview advice, newsletters, podcasts, and lists of the companies building the tools that data engineers use every day.

The handbook is closely tied to two companion websites. The DataEngineer.io blog publishes weekly newsletters and deep-dive articles written by practitioners, while learn.dataexpert.io hosts the structured bootcamp programs that pair with the repository content. Together these three surfaces form a complete learning ecosystem: the handbook for curation and reference, the blog for ongoing education, and the learning platform for guided practice. Because everything is open and free at the point of access, the only barrier to entry is the time and curiosity you bring.

How It Works

The diagram above captures the structure of the handbook as a learning roadmap rather than a static list of links. Reading it from top to bottom, you can see how a learner flows through the resource.

At the very top sits the Learner node, which branches into three proficiency tiers: Beginner, Intermediate, and Advanced. These tiers are colored green to signal that they represent the input or human side of the system. The arrows between them, labeled starts, progress, and master, make the progression explicit. A newcomer does not jump straight into distributed systems design. They start with the beginner bootcamp, progress through intermediate topics like dimensional modeling and Spark, and only then tackle advanced subjects such as streaming with Flink or system design interviews.

The middle band, colored blue, holds the four core resource categories that the handbook curates: Books, Bootcamps, Projects, and Interviews. These are the workhorses of the learning journey. The edges show how each learner tier connects to the most relevant core resource.

A Beginner is directed first to the Bootcamps, where the free four-week program teaches SQL, Python, and pipeline fundamentals. An Intermediate learner is pointed at the Books list, where titles like Fundamentals of Data Engineering and Designing Data-Intensive Applications fill in theoretical depth, and at the Projects section, where end-to-end builds turn theory into muscle memory. An Advanced learner is routed to the Interviews section to prepare for system design, data modeling, and data architecture rounds at top companies.

The bottom-left cluster, colored orange, groups the Tools and Companies that the handbook catalogues. The handbook does not just name tools in isolation. It organizes them into categories such as Orchestration (Airflow, Dagster, Prefect), Data Lake (Databricks, Iceberg), and Data Warehouse (Snowflake).

The dashed edges from the Projects node into these tool categories communicate a key idea: the best way to learn a tool is to use it inside a real project. Rather than reading documentation in a vacuum, you pick a project from the projects list and reach for the tools that the project demands. Each tool category then fans out into the specific products, so Orchestration connects to Airflow, Dagster, and Prefect, while Data Lake connects to Databricks and Iceberg.

The bottom-right cluster, colored purple, represents the Community and Career dimension. Here you find Discord servers, Slack groups, newsletters, and a Career Growth node. The dashed purple edges flowing back upward toward the learner tiers illustrate that community support is not a one-time stop. It is a continuous feedback loop.

Beginners get unblocked in the DataExpert.io Discord, intermediate learners exchange ideas in the Data Talks Club Slack, and advanced practitioners stay current through newsletters like Data Engineering Weekly. Finally, the Interviews node feeds into Career Growth, which in turn reinforces the Advanced tier, closing the loop from learning to practicing to growing.

The color scheme is deliberate and consistent throughout the diagram. Green marks anything that is human input, namely the learner and their progression stages. Blue marks the curated core resources that form the spine of the handbook. Orange marks the tools and companies that learners will eventually work with in production. Purple marks the social and career layer that sustains long-term growth.

The orthogonal routing keeps every connector at clean right angles so that even a dense graph stays readable. The white text halos ensure that edge labels such as enroll, study, build, and leads to remain legible regardless of the background they cross. The layout respects a maximum of eight blocks per row, which keeps the diagram scannable even as the number of curated tools and communities grows over time.

Taken together, the diagram communicates the central thesis of the handbook: data engineering is not a single skill but a connected journey. The resources are not isolated bookmarks. They feed into one another, with books informing projects, projects requiring tools, tools fitting into company stacks, and communities supporting every step. A learner who follows the arrows from the green Learner node down through the blue core resources and out into the orange tools and purple communities will, by construction, cover the same ground that working data engineers cover in their first few years on the job.

Learning Path

The handbook recommends a phased approach rather than a rigid sequence. The 2024 breaking into data engineering roadmap published on the DataEngineer.io blog is the canonical starting point for newcomers. It frames the journey as three overlapping stages, and the repository mirrors this structure with dedicated folders and files for each stage.

The first stage is foundations. Here the focus is on SQL fluency, Python programming, and an understanding of how data moves from a source system into a destination. The four-week beginner bootcamp is designed for exactly this stage. Learners install the required software, follow the introduction materials, and build their first simple pipelines. The goal is not mastery but familiarity. You should be able to write a join, load a CSV into a database, and explain what a fact table is before moving on.

The second stage is depth and breadth. The six-week intermediate bootcamp takes over, covering dimensional data modeling, fact table design, Spark fundamentals, Apache Flink, analytical patterns, KPIs, experimentation, and pipeline maintenance. In parallel, the books list provides the theoretical scaffolding. Designing Data-Intensive Applications explains the distributed systems concepts that underpin every modern data tool, while Fundamentals of Data Engineering gives a vendor-neutral view of the entire lifecycle.

The third stage is application and interview readiness. The projects section offers end-to-end builds, such as the popular Uber data engineering project with BigQuery, that let you assemble everything you have learned into a portfolio piece. The interviews section then sharpens your ability to communicate that knowledge under pressure, with dedicated material on the SQL interview, the data modeling interview, the data architecture interview, and the data structures and algorithms interview. The path is not strictly linear. Many learners cycle between projects and books as they encounter gaps, and the community channels exist precisely to help you decide what to study next.

Books

The handbook curates a list of more than 25 books, and three titles are highlighted as must-reads. Fundamentals of Data Engineering by Joe Reis and Matt Housley provides a vendor-neutral framework for thinking about the data lifecycle, from ingestion to serving. Designing Data-Intensive Applications by Martin Kleppmann is the canonical reference for the distributed systems concepts that every senior data engineer is expected to understand, including replication, partitioning, transactions, and consensus. Designing Machine Learning Systems by Chip Huyen bridges data engineering and ML engineering, covering the infrastructure that production ML systems require.

Beyond the top three, the list spans the entire stack. For warehousing and modeling there is Kimball’s Data Warehouse Toolkit and Data Mesh by Zhamak Dehghani. For streaming there is Streaming Systems and Stream Processing with Apache Flink. For the Spark ecosystem there is Spark: The Definitive Guide, Learning Spark, and High Performance Spark. For the lakehouse pattern there are dedicated books on Delta Lake, Apache Iceberg, and Trino. For practitioners who want breadth there is 97 Things Every Data Engineer Should Know and Data Engineering Design Patterns. The list also includes practical titles like Data Engineering with dbt, Snowflake Data Engineering, and Unlocking dbt. Reading even a fraction of these books gives you a vocabulary that maps directly to the systems you will operate in production.

Communities

Learning data engineering in isolation is slow. The handbook curates more than 10 communities, and three are highlighted as essential. The DataExpert.io Community Discord is the most directly tied to the bootcamps and the repository itself. It is where bootcamp cohorts coordinate, where learners ask questions, and where Zach Wilson and other practitioners occasionally share insights. The Data Talks Club Slack is home to the well-known Data Engineering Zoomcamp and a broader community that spans analytics and ML. The Data Engineer Things Community offers articles, a newsletter, and a gathering place focused on practical data engineering.

The list also includes communities focused on adjacent areas that data engineers should be aware of. The AdalFlow Discord and the Chip Huyen MLOps Discord cover the ML and LLM application side. The dbt Community is the central hub for analytics engineers. Reddit communities such as r/dataengineering, r/databricks, and r/MicrosoftFabric provide ongoing discussion and news. The Microsoft Fabric Community offers official support for that platform. Joining even one or two of these communities and participating consistently is one of the highest-leverage activities a learner can undertake, because it exposes you to the real problems practitioners are solving today.

Tools and Companies

One of the most valuable aspects of the handbook is its organized catalogue of companies and tools. Rather than presenting an undifferentiated list, the repository groups vendors by the layer of the data stack they occupy. This taxonomy is itself a learning aid, because it teaches you how the modern data stack fits together.

For orchestration, the handbook lists Apache Airflow, Dagster, Prefect, Mage, Astronomer, Kestra, Shipyard, and Hamilton. These are the schedulers and workflow engines that define when and how pipelines run. For the data lake and storage layer, the list includes Databricks, Apache Iceberg, Delta Lake, Tabular, Onehouse, Microsoft, Apache Polaris, Lakekeeper, Ilum, and DuckLake. These are the technologies that store and table massive datasets. For the data warehouse layer, the handbook points to Snowflake, Firebolt, and Databend as the analytical query engines.

The catalogue continues across data quality (dbt, Great Expectations, Metaplane, Soda, DQOps), data integration (Fivetran, Airbyte, dlt, Meltano, Estuary, Sling), analytics and visualization (Tableau, Power BI, Looker Studio, Metabase, Apache Superset, Preset, Hex, Evidence, Redash, Lightdash), modern OLAP (ClickHouse, Apache Druid, Apache Pinot, DuckDB, StarRocks, QuestDB), real-time data (RisingWave, Striim, Responsive, Aggregations.io), semantic layers (Cube, dbt Semantic Layer), data lineage (OpenLineage), and education companies (DataExpert.io, ByteByteGo, AlgoExpert, LearnDataEngineering.com). The repository also links to the engineering blogs of Netflix, Uber, Databricks, Airbnb, AWS, Microsoft, Oracle, and Meta, so you can read first-hand how these companies operate their platforms. Together this catalogue is a map of the industry, and studying it is itself an education in how data systems are built and sold.

Bootcamps

The handbook ships with two free bootcamps that map directly to the beginner and intermediate tiers shown in the diagram. The four-week beginner bootcamp is designed for people with little to no prior experience. It begins with an introduction that sets expectations and a software setup guide that gets your local environment ready. From there the weeks cover SQL basics, Python fundamentals, and the construction of a first pipeline. The materials live in the beginner-bootcamp folder of the repository, with introduction.md and software.md as the entry points.

The six-week intermediate bootcamp is far more substantial and is one of the standout offerings of the entire handbook. Its materials folder contains six modules, each with lecture labs, homework, and supporting code. The first module covers one-dimensional data modeling, including slowly changing dimensions and idempotent pipelines, with a Docker-based PostgreSQL environment and SQL scripts you can run immediately. The second module tackles fact data modeling with cumulative tables and array metrics. The third module introduces Spark fundamentals through PySpark notebooks and a tested job structure. There is also an Apache Flink training module that includes a streaming aggregation job.

The remaining intermediate modules cover applying analytical patterns (funnel analysis, growth accounting, retention, window-based analysis), KPIs and experimentation with a small server implementation, and data pipeline maintenance with runbooks and a real-world growth pipeline example. There is even a Databricks AI bootcamp that spans three days and covers Lakebase, context engineering with vector databases, and agent-based end-to-end AI applications. The following snippet shows the typical folder layout of an intermediate module, so you can see how the materials are structured for hands-on learning:

intermediate-bootcamp/
  materials/
    1-dimensional-data-modeling/
      README.md
      docker-compose.yml
      Makefile
      lecture-lab/
        analytical_query.sql
        incremental_scd_query.sql
        pipeline_query.sql
      homework/
        homework.md
      scripts/
        init-db.sh
      sql/
        players.sql
        game_details.sql

Because each module is self-contained and version-controlled, you can clone the repository and work through any module at your own pace. The homework files give you concrete exercises, and the lecture labs give you reference implementations to compare against. This combination of structure, real datasets, and tested code is what separates the handbook from a simple link list.

Conclusion

The Data Engineering Handbook earns its name by being a genuine single source of truth for anyone serious about the field. It does not try to teach you everything itself. Instead, it curates the best books, free bootcamps, hands-on projects, interview prep, tools, companies, and communities, and organizes them into a coherent path that takes you from beginner to advanced. The diagram at the top of this post captures that path visually, and each section of this guide expanded on one band of that diagram.

If you are just starting out, the recommended sequence is to join the beginner bootcamp, pick one book from the must-read list, and join the DataExpert.io Discord. From there, work through the intermediate bootcamp, build one project from the projects list, and begin interview preparation. The handbook will be there at every step, pointing you to the next resource. The repository is open source and actively maintained, so as the data landscape evolves, the curated lists evolve with it. Clone it, star it, and treat it as the reference it is. The most reliable way to become an amazing data engineer is to follow a path that others have already validated, and that is exactly what the Data Engineering Handbook provides.

Enjoyed this post? Never miss out on future posts by following us