XGBoost has been the default answer for tabular machine learning for the better part of a decade, and unlike many tools that survive by inertia, its source code is still worth reading in 2026. The repository is really two programs living together: a C++ core that is compiled into the libxgboost shared library, and a Python package that is, in large part, a polite way of calling into that library. The gradient-boosted-tree algorithm itself — residuals, histograms, split finding — lives almost entirely in src/, while the estimator surface you actually type into lives in python-package/xgboost/. Understanding where the seam sits, and what crosses it, explains a lot of XGBoost’s behavior: why parameters have such precise names, why custom objectives look the way they do, and why a Python object called Booster is something distinct from the estimator you got from XGBRegressor.

This tour follows one training call from the top of the stack to the bottom and back. We start at fit() in the scikit-learn wrapper, descend through train() and Booster.update, cross the C ABI, land in Learner — the orchestrator that owns the model state — and then follow the gradient container into GBTree, the concrete gradient booster, and finally into the histogram tree grower where OpenMP workers scatter gradient pairs into bins. Then we come back up the other side: prediction is not simply the mirror image of training, because the prediction subsystem was restructured into a kernel-dispatch design that registers CPU and CUDA implementations separately.

Everything cited below is verified against the actual master branch; every file path and line number in the diagrams was checked against the checked-out tree before rendering. XGBoost moves files around between releases more often than most projects, so if you are reading this on a different version, expect drift — but the architecture has been stable for years, and the two diagrams we generated capture it at commit level.

The overview: from the Python estimator surface, through the ctypes boundary and the C ABI, into the Learner, the GBTree booster, and the histogram-based tree grower, flanked by the objective and metric registries.

Reading the overview from left to right:

• The scikit-learn surface is one file with a few classes. XGBModel.fit, L1336 normalizes inputs and delegates to the functional API, while XGBClassifier, L1751 and XGBRegressor, L2052 only add label handling, class counting, and prediction post-processing on top.
• The training loop is train, L53 — a genuine Python for loop, not a C++ iteration hidden behind an iterator. Each round calls bst.update and then gives callbacks a chance to stop the run.
• Booster, L1794 in core.py is the model object proper: it wraps an opaque native handle and forwards update, predict, and serialization calls to it. Everything else in the Python package is convenience around this class.
• The crossing into native code happens through _LIB, L184, a ctypes handle loaded in _c_api.py, with _check_call, L187 translating integer error codes into exceptions.
• On the native side, XGBoosterUpdateOneIter, L1146 is the workhorse entry point; it does nothing clever and simply forwards to the Learner object hidden behind the handle.
• LearnerImpl, L970 is the brain: it lazily builds the objective, the booster, and the metrics, then executes one training iteration end to end.
• The heavy lifting is owned by GBTree::DoBoost, L186, which turns a container of gradient pairs into new trees by invoking registered TreeUpdater::Create, L17 implementations.

From Install to Your First Model

pip install xgboost gives you a wheel that contains both halves of the system: the prebuilt libxgboost binary and the Python package around it. The first interesting thing happens when you call fit(). Despite the scikit-learn costume, nothing here reimplements gradient boosting in Python — XGBModel.fit, L1336 converts your data into a DMatrix, L681, which is the canonical native dataset container, and then hands everything to train. If your data is dense enough and you want to skip the copy, QuantileDMatrix (L1449) builds the quantized bins up front and shares them with training, which is the same trick the GPU path has used for years.

The loop inside train, L53 is short enough to read in one screen: it constructs a Booster, L1794 with the training DMatrix plus any evaluation sets, wraps user callbacks in a CallbackContainer, and then runs for i in range(num_boost_round), calling bst.update(dtrain, iteration=i, fobj=obj) at line 203 and checking after_iteration right after. Early stopping is not special-cased anywhere in the loop — it is just a callback that watches the evaluation history. The stock callbacks live in callback.py: TrainingCallback, L52 defines the interface, and EarlyStopping, L312 implements the patience logic you rely on every day.

This design is a good contrast with the tensor-graph systems we covered in the PyTorch and scikit-learn tours: XGBoost keeps the iteration in Python where it is trivially interruptible, and pushes only per-round computation down into C++.

Crossing the ABI, the native side mirrors the Python objects one level down. XGBoosterCreate, L1036 allocates a Learner with the cached DMatrix handles; XGBoosterUpdateOneIter then calls into it. Dataset construction follows the same route: adapters for NumPy arrays, SciPy matrices, cuDF frames, and external iterators are all funneled through DMatrix::Create (DMatrix::Create, L981), which decides whether to materialize an in-memory SimpleDMatrix, L185 or set up one of the paged, out-of-core variants.

The Learner: One Iteration, Three Factories

src/learner.cc is one of the largest translation units in the project, and it is organized as a stack of classes that each add one responsibility: LearnerModelStateContainer (line 238) owns the model, LearnerConfiguration (line 493) absorbs the keyword-argument soup, LearnerIO (line 816) handles serialization, and LearnerImpl, L970 finally implements the training behavior. Configuration is lazy — parameters are buffered and consumed at first use, which is why you can set booster after construction.

The heart is UpdateOneIter, L1057, and its sequence is the algorithm in miniature. First it calls PredictRaw on the training set to get current margin values. Then GetGradient applies the objective’s loss transform to produce a gradient/hessian pair for every row. Then, at line 1082, those gradients go to gbm_->DoBoost and the model grows by one step. Evaluation follows the same shape: EvalOneIter, L1106 predicts on each evaluation set, runs metrics, and formats the familiar train-rmse:0.31 table.

The three factories that LearnerImpl consults during configuration are worth knowing by name because they are the extension points of the whole system. ObjFunction::Create, L19 looks up a name like reg:squarederror in the dmlc registry and returns a polymorphic loss; the actual math for, say, logistic regression is a small header of structs such as LogisticPredTransform (L41) that convert margins to probabilities and back to gradients. Metric::Create, L45 does the same for rmse, auc, and friends, and GradientBooster::Create, L20 selects between gbtree and gblinear. A custom objective you pass from Python does not register anywhere — it takes the alternate route, where the learner skips GetGradient and you supply gradients yourself through BoostOneIter (L1086), exposed to Python as XGBoosterBoostOneIter (L1157).

How a Tree Actually Grows

GBTree (src/gbm/gbtree.cc) is the concrete booster that owns the ensemble. Its BoostNewTrees, L323 method creates fresh empty trees for this round, rescales the learning rate across groups, and then loops over its updaters at line 359 calling up->Update(...). The updaters themselves were built earlier by TreeUpdater::Create — the construction site is at L435 — which is the same registry pattern as objectives and metrics. For the default CPU configuration, the name that resolves is grow_quantile_histmaker, implemented by QuantileHistMaker, L563.

Its Update method (line 589) delegates to UpdateTree, L92, a free function that is the clearest 60 lines in the repository. It seeds a Driver, L58 — a priority queue whose comparator switches between depth-wise and loss-guided growth via DepthWise (L24) and LossGuide (L29), which is exactly what the grow_policy parameter selects. Then it runs the expansion loop: take the root entry from HistUpdater::InitRoot, L475, pop the best candidates, call ApplyTreeSplit (L471) to write the split into the tree, reposition rows through UpdatePosition, build histograms for the children, and evaluate them.

The histogram machinery underneath is where XGBoost earns its speed. Features are pre-binned into a GHistIndexMatrix, L20, so split finding never compares raw feature values — it scans integer bin indices. Those bins come from cut points computed by a weighted quantile sketch: WQuantileSketch, L565 maintains mergeable summaries of each feature’s distribution, and AddCutPoints (L526) turns merged summaries into the final bin boundaries. During growth, HistogramBuilder, L44 accumulates gradient pairs per node through common::BuildHist (L499), an OpenMP scatter over row indices. Split evaluation then scans those histograms — EvaluateSplits in HistEvaluator (L267) computes the gain for every bin of every feature and returns a CPUExpandEntry, L29 per candidate node. After a split is applied, CommonRowPartitioner (L32) moves row indices to their child nodes so the next round of histograms can be built incrementally.

The GPU variants under src/tree/gpu_hist/ mirror this entire structure with CUDA kernels — same updater interface, same entry types, different execution — which is why parameters like max_bin behave identically across devices.

Prediction: Kernel Dispatch

Prediction used to be organized as a Predictor registry with CPU and GPU implementations, but current master replaced that with a kernel-dispatch design that is closer to how the CUDA internals already worked. GBTree::PredictBatch, L653 calls common::DispatchKernel<predictor::PredictBatchKernel> at line 706; the kernels themselves are declared in prediction_kernel.h, L31 as templated stubs with CPU and CUDA registrations — PredictBatchKernel, InplacePredictKernel (L43), leaf/contribution/SHAP variants each have their own dispatch target. On the Python side this is visible as Booster.predict, L2424 versus the lower-level inplace_predict, L2561, which skips DMatrix construction entirely and walks trees directly over your array data. That second path is the one the scikit-learn wrapper uses for fast predict() calls, and it is often an order of magnitude faster on small batches.

For deployment, this layer matters more than the training side: a serialized model is just trees and configuration, which is why export targets — ONNX converters and similar — only need the model schema, not the trainer. If you are comparing inference runtimes for boosted-tree models, our ONNX and ONNX Runtime tours cover what happens after XGBoost hands the model off.

Reading the Detail Diagram

The detail diagram expands each block of the overview into its real files and call sites, including the data preparation column and the prediction kernels that the overview compresses into the booster.

The detail view: 33 nodes across eight groups, from the ctypes `_LIB` shim down to the OpenMP histogram scatter, with the row partitioner and quantile sketch paths made explicit.

Three paths are worth tracing. The training path runs XGBModel.fit → train → Booster.update → _LIB → XGBoosterUpdateOneIter → UpdateOneIter → GBTree::DoBoost → BoostNewTrees → the updater loop — every arrow on that chain is a real function call you can put a breakpoint on. The data path shows why the first call to fit() is slow: adapter conversion, DMatrix::Create, and the quantile sketch that computes cuts all happen once, cached, before the first tree grows. The prediction path branches out of GBTree::PredictBatch into the kernel table, which is deliberately disconnected from the growth machinery — training and inference share the model, not the code path.

Honest Limits

What this tour does not cover: the distributed stack. src/collective/ — communicators, allreduce, the tracker — is a self-contained subsystem for multi-node training that we skipped entirely, and the federated and external-memory variants add further layers. The CUDA half of the tree builder is also only sketched here; src/tree/gpu_hist/ is a full parallel implementation with its own row partitioner and quantizers, and its performance characteristics deserve their own post. The JVM packages and the R package are bindings at the same ABI we crossed, but their parameter plumbing differs in details that matter.

Some limitations are inherent to the design rather than to our coverage. The Python training loop is a loop: if you need to fuse ten thousand boosting rounds without host round-trips, the per-iteration callback overhead is real, though the C++ side amortizes most of it. The lazy configuration in the learner is convenient but makes some parameter errors surface late, on the first iteration, deep inside a stack trace that crossed three layers of factory. And the histogram approximation means the algorithm is sensitive to max_bin in ways the exact greedy grower (grow_colmaker, also still in the tree) is not — a knob that changes both speed and the trees themselves. Finally, this codebase churns: the predictor registry became kernel dispatch, LearnerModelState split out of the learner, and multi-target trees arrived as a distinct tree model. If you read the source in two years, some line numbers will have moved — the shapes, though, have been remarkably durable.

Enjoyed this post? Never miss out on future posts by following us