spaCy needs little introduction. It is the industrial-strength natural language processing library from Explosion, written in a deliberate mix of Python and Cython, and it has become the default choice when you need tokenization, part-of-speech tagging, dependency parsing, named entity recognition, or text classification that actually runs fast in production. Unlike scikit-learn, which treats text as a bag of features for classical estimators, spaCy is opinionated about document structure: every text becomes a Doc object, and a configured stack of processing components annotates that object in a fixed order. If you have read our Tokenizers post, you saw how byte-level BPE breaks text into subwords; spaCy answers a different question — how to turn raw text into linguistically structured objects a model can learn from and an application can act on.

The source is worth reading because the performance story is not a wrapper around something else. Files like spacy/tokenizer.pyx, spacy/tokens/doc.pyx, and spacy/ml/parser_model.pyx are Cython that compiles down to C, and the whole design follows a simple discipline: keep the hot path in compact structs, keep orchestration in thin Python. At the same time, the neural side is built on Thinc, Explosion’s own functional deep-learning library, so model wiring is declared in config files and resolved through a registry rather than hand-coded class hierarchies. That combination — Cython data structures plus registry-driven Thinc models — is what makes spaCy feel different from both classic ML libraries and the Transformers ecosystem.

In this tour we walk the repository the way a request flows through it: the Language engine that owns everything, the tokenizer that produces the Doc, the Doc object model that every component shares, the Thinc models underneath tagging, parsing and entity recognition, and finally the training loop that ties them together. Two diagrams anchor the tour — a compact overview and a deeper detail map — and every symbol we mention is linked to the real source on GitHub so you can follow along in the tree.

The spaCy overview: one Language engine owns the tokenizer and component stack, Thinc models do the learning, and every path ends at the shared Doc object.

Reading the overview from left to right:

• [Language](https://github.com/explosion/spaCy/blob/master/spacy/language.py#L138) is the nlp object. [add_pipe](https://github.com/explosion/spaCy/blob/master/spacy/language.py#L768) registers a processing component by name, [__call__](https://github.com/explosion/spaCy/blob/master/spacy/language.py#L1020) runs the whole stack over a text, and [make_doc](https://github.com/explosion/spaCy/blob/master/spacy/language.py#L1114) hands off to the tokenizer.
• The [Tokenizer](https://github.com/explosion/spaCy/blob/master/spacy/tokenizer.pyx#L25) is pure Cython with no neural component at all. Its [__call__](https://github.com/explosion/spaCy/blob/master/spacy/tokenizer.pyx#L152) applies prefix, infix, and suffix rules plus an optional exception table; a batched [pipe](https://github.com/explosion/spaCy/blob/master/spacy/tokenizer.pyx#L212) exists for bulk work.
• [Vocab](https://github.com/explosion/spaCy/blob/master/spacy/vocab.pyx#L46) centralizes string storage, lexeme metadata, and vector tables so that one nlp instance shares a single hash-to-string map across all documents.
• [Doc](https://github.com/explosion/spaCy/blob/master/spacy/tokens/doc.pyx#L130) is the container every component reads and writes: the tokenizer fills it, taggers and parsers annotate it, and [set_ents](https://github.com/explosion/spaCy/blob/master/spacy/tokens/doc.pyx#L800) turns character-level predictions into entity spans; [noun_chunks](https://github.com/explosion/spaCy/blob/master/spacy/tokens/doc.pyx#L888) exposes one of the derived views.
• [Tok2Vec](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py#L102) builds the shared token encoder — [MultiHashEmbed](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py#L122) for hashed embeddings, then a [MaxoutWindowEncoder](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py#L275) or [BiLSTMEncoder](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py#L328) for context.
• [ParserStepModel](https://github.com/explosion/spaCy/blob/master/spacy/ml/parser_model.pyx#L236) is the transition-based step predictor shared by the dependency parser and the entity recognizer — one Cython model, two grammars of actions.
• The rule-based side lives in [Matcher](https://github.com/explosion/spaCy/blob/master/spacy/matcher/matcher.pyx#L32): [add](https://github.com/explosion/spaCy/blob/master/spacy/matcher/matcher.pyx#L79) registers token patterns with optional [on_match](https://github.com/explosion/spaCy/blob/master/spacy/matcher/matcher.pyx#L214) callbacks, no training data required.
• [train](https://github.com/explosion/spaCy/blob/master/spacy/training/loop.py#L35) drives optimization by calling nlp.update, and [train_while_improving](https://github.com/explosion/spaCy/blob/master/spacy/training/loop.py#L153) adds early stopping on a dev set.
• [load_model](https://github.com/explosion/spaCy/blob/master/spacy/util.py#L492) in spacy/util.py resolves a package name, directory, or shortcut into a fully assembled Language object.

From Install to Your First Model

Getting started is deliberately short:

pip install spacy
python -m spacy download en_core_web_sm

Then in Python:

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("PyShine ships fast text analytics.")
print([t.text for t in doc])
print([(e.text, e.label_) for e in doc.ents])

Three things happen inside that innocent spacy.load. First, [load_model](https://github.com/explosion/spaCy/blob/master/spacy/util.py#L492) locates the installed package and reads its config.cfg, a serialized description of the tokenizer settings and every component with its factory and hyperparameters. Second, that config is resolved through the registry — the same mechanism you will meet on the Thinc side — so each component is constructed by a registered factory function rather than a hardcoded import. Third, the returned Language object wires the components in the declared order; at runtime nlp(text) tokenizes once and then lets each component annotate the shared Doc in sequence.

This config-first design is why spaCy models are inspectable. Call nlp.analyze_pipes() and the library reports, per component, which attributes it sets and requires — the implementation is [analyze_pipes](https://github.com/explosion/spaCy/blob/master/spacy/pipe_analysis.py#L81), a small module that earns its place by catching ordering mistakes like a lemmatizer placed before a tagger. The same discipline powers training: [init_nlp](https://github.com/explosion/spaCy/blob/master/spacy/training/initialize.py#L35) builds a fresh Language from config, and only afterwards does data touch the model. Nothing about your first model was special-cased; it was assembled from the same parts you would assemble for a custom one.

The Doc Object Model: One Array, Many Views

The deepest design decision in spaCy is the Doc. Its constructor at [__init__](https://github.com/explosion/spaCy/blob/master/spacy/tokens/doc.pyx#L202) allocates flat C arrays — one per annotation field — sized to the token count. A [Token](https://github.com/explosion/spaCy/blob/master/spacy/tokens/token.pyx#L44) is not a Python object holding its own data; it is a lightweight view with an index into those arrays, created on access. A [Span](https://github.com/explosion/spaCy/blob/master/spacy/tokens/span.pyx#L24) is the same trick for a token range. Iterating a document therefore costs no object churn: you are reading typed views over memory the document already owns.

This is also why the tokenizer can be so fast. Because Doc storage is preallocated and the string interning lives in [Vocab](https://github.com/explosion/spaCy/blob/master/spacy/vocab.pyx#L46), splitting a sentence is just pointer arithmetic and hash lookups. The vocabulary pairs each string with a [Lexeme](https://github.com/explosion/spaCy/blob/master/spacy/lexeme.pyx#L43) — cached flags like is_alpha, is_stop, and language-specific attributes — while [Morphology](https://github.com/explosion/spaCy/blob/master/spacy/morphology.pyx#L14) compresses feature bundles into dense slot tables and [Lookups](https://github.com/explosion/spaCy/blob/master/spacy/lookups.py#L162) holds data tables such as lemma dictionaries loaded from the language data directory.

The practical consequence: annotations are decoupled from algorithms. Entity boundaries set via [set_ents](https://github.com/explosion/spaCy/blob/master/spacy/tokens/doc.pyx#L800), dependency arcs, morphological features, and custom extension attributes all live as parallel arrays on the same document, so any component can consume any other component’s output without a special interface. It also makes serialization uniform — a Doc and its annotations survive a round trip through DocBin unchanged, which is exactly what the training data machinery relies on.

Extension attributes deserve a mention too, because they show how far the view idea is pushed. User code can attach its own per-token, per-span, or per-document values through the underscore module, and those custom attributes ride along in serialization like built-in ones. Retokenization — merging multi-token expressions into one or splitting a token in two — is implemented as a surgical array splice on the same document, so a rule-based matcher and a merge step can cooperate without rebuilding anything. Both facilities exist precisely because the document is a data structure rather than a bag of Python objects.

The detail map: Doc storage and its views on one side, the Thinc model zoo and the transition-based step model in the middle, matchers, training and rendering utilities around them.

Under the Hood: Thinc Models and Transition Systems

Every trainable component in spaCy is a thin shell around a Thinc Model. The shared encoder pattern shows up first: [build_Tok2Vec_model](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py#L102) composes hashed embedding tables with a context encoder, and downstream components subscribe to its output with the listener mechanism — see [tok2vec_listener_v1](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py#L31), which lets a component receive gradients from another component’s encoder instead of keeping a private copy. The tagger is the minimal example: [build_tagger_model](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tagger.py#L9) wires encoder output to a softmax over tag classes. Text classification varies the head instead of the encoder — [build_simple_cnn_text_classifier](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/textcat.py#L46) pools token vectors, while [build_textcat_parametric_attention_v1](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/textcat.py#L218) learns how much each token should matter.

Structure prediction goes through transition systems. [build_tb_parser_model](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/parser.py#L13) assembles the stacked architecture, and the actual state machine lives in Cython: [ParserStepModel](https://github.com/explosion/spaCy/blob/master/spacy/ml/parser_model.pyx#L236) scores the next action for a batch of partial parses, with [precompute_hiddens](https://github.com/explosion/spaCy/blob/master/spacy/ml/parser_model.pyx#L368) hoisting the expensive encoder pass out of the per-step loop. The entity recognizer reuses this machinery with a different action set — which is why tagging, parsing, and NER all feel consistent once you have read one of them.

More specialized heads follow the same recipe. Span categorization builds ragged span representations in [build_spancat_model](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/spancat.py#L46) and scores candidates against configurable length ranges. Entity linking adds retrieval to the mix: [build_nel_encoder](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/entity_linker.py#L30) encodes mention context and [load_kb](https://github.com/explosion/spaCy/blob/master/spacy/ml/models/entity_linker.py#L93) pulls the knowledge-base side, so disambiguation becomes a similarity problem against candidate identifiers.

Training, Batching, and Evaluation

Training data is expressed as Example pairs. The Cython class at [Example](https://github.com/explosion/spaCy/blob/master/spacy/training/example.pyx#L78) holds a predicted Doc and a reference Doc plus an alignment between them, so losses can be computed token-aligned even when tokenization differs. Converters bring external corpora into this format — [conllu_to_docs](https://github.com/explosion/spaCy/blob/master/spacy/training/converters/conllu_to_docs.py#L11) handles CoNLL-U treebanks, which is the bridge from academic datasets to spaCy training, and the rest of the Datasets world plugs in through the same entry points.

The loop itself is refreshingly small. [train](https://github.com/explosion/spaCy/blob/master/spacy/training/loop.py#L35) alternates shuffling, batching, nlp.update, and evaluation; [minibatch_by_padded_size](https://github.com/explosion/spaCy/blob/master/spacy/training/batchers.py#L95) groups examples of similar padded length so tensor work is not wasted on padding. Evaluation funnels through the [Scorer](https://github.com/explosion/spaCy/blob/master/spacy/scorer.py#L111), which aggregates token-level, span-level, and text-level metrics into one report, and [train_while_improving](https://github.com/explosion/spaCy/blob/master/spacy/training/loop.py#L153) stops when the dev score plateaus. For debugging outputs visually, the renderers in spacy/displacy/render.py — [SpanRenderer](https://github.com/explosion/spaCy/blob/master/spacy/displacy/render.py#L52), [DependencyRenderer](https://github.com/explosion/spaCy/blob/master/spacy/displacy/render.py#L291), and [EntityRenderer](https://github.com/explosion/spaCy/blob/master/spacy/displacy/render.py#L504) — turn the same Doc annotations into SVG and HTML.

Honest Limits

The small pretrained models that most people start with trade accuracy for speed; for competitive numbers you move to transformer-based weights, at a real cost in memory and latency. Tokenization is rule-based per language: excellent for well-formed text, but domain-specific text (social media, code, historical orthography) often needs custom exceptions or a different tokenizer altogether, as our earlier tokenizer survey shows.

Architecturally, the component stack is sequential and unidirectional — a component sees the document only after earlier components have finished, and sharing beyond the tok2vec-listener mechanism is limited, so genuinely interactive multi-task inference is awkward to express. Training requires gold data in the Example format; there is no first-class path from raw unlabeled text to a custom component short of pretraining tricks. Entity linking is powerful but presumes you can build and host a knowledge base. Finally, the Cython layer means you should not expect to edit a .pyx file and rerun — a C compiler toolchain and the build steps from the repository are required, which raises the bar for casual contribution compared to pure-Python libraries. None of these are hidden; they are the visible trade-offs of a library optimized to be fast and predictable in production, and knowing where the C ends and the Python begins is the key to extending it confidently.

Read this way, the repository is less a black box than a layered argument: compact Cython structures where speed matters, a registry-driven model layer where flexibility matters, and a small orchestration core that keeps both honest.

Enjoyed this post? Never miss out on future posts by following us