Most search tutorials stop at the API surface. This one goes the other way: we open the crate itself. Tantivy is a fast full-text search engine library written in Rust — closer in spirit to Apache Lucene than to a turnkey server like Elasticsearch, in the words of its own README. It is the engine underneath Quickwit, and ParadeDB ships it inside Postgres for pg_search. The crate is at version 0.27.0 on the main branch; every file path and line number below was verified against that tree with grep before being linked.

Two diagrams accompany the tour. Both were generated from the actual repository layout rather than from memory, and both are clickable — each node links to the exact source file on GitHub. The first is a high-level pass over the whole system; the second zooms into seven subsystems: schema, core objects, the indexer, storage, querying, the inverted index, and tokenizers. If you have ever wondered what actually happens between add_document and a scored hit list, the answer is a chain of data structures with wonderful Lucene-flavored names: postings, term dictionaries, field norms, fast fields, and the doc store.

A caution before we start: Tantivy is a library, not a service. There is no cluster and no REST endpoint. The interesting complexity is entirely in the on-disk data structures and the commit protocol that keeps readers consistent while writers mutate the index. That is exactly the layer these diagrams try to make visible.

Overview: documents flow from a frozen schema through the multithreaded writer into immutable segment files, while the read side reloads those files into Searcher snapshots. Every node links to its source file on the main branch.

Reading the overview from left to right:

• Everything starts with SchemaBuilder. You declare fields with typed options — add_text_field at line 163, then build() at line 244 freezes an immutable Schema. Each field becomes a FieldEntry (src/schema/field_entry.rs) carrying a FieldType — text, u64, f64, date, facet, bytes, or JSON.
• The Index struct pairs that schema with a storage location. create_in_ram (line 444) and open (line 650) are the two ways in; reader() at line 590 and writer() at line 753 hand out the two halves of the system.
• On the write side, IndexWriter is the only door in. add_document (line 741) routes each document to one of several indexing threads, delete_term (line 694) enqueues a tombstone, and commit (line 678) makes everything visible atomically.
• Each thread owns a SegmentWriter that buffers a batch in memory: it tokenizes text, fills the in-memory postings, store, fast fields and field norms, and only then seals the batch to disk.
• At commit time the SegmentManager takes over — it tracks which segments are live, applies queued deletes, and consults LogMergePolicy about when to compact.
• Merges run in background threads through the IndexMerger, which streams several segments into one while dropping deleted documents along the way.
• On the read side, IndexReader watches for new commits and searcher() at line 315 vends an immutable Searcher snapshot.
• Finally, a QueryParser turns title:sea AND body:whale into a query tree, and search() at line 180 pushes every match into a collector such as TopDocs.

The detailed view: 34 code-level nodes across seven subsystems, from the FST-backed term dictionary down to the bitpacked posting blocks.

Why an index is a forest of immutable segments

The single most important design fact in Tantivy is that an index is a collection of small, independent, immutable segments. Each segment is identified by a UUID-based SegmentId and owns its own copy of every data structure. A tiny meta.json records the live segment list plus the schema, and commit swaps that file atomically. SegmentMeta carries the bookkeeping: doc counts, max doc, whether the segment has deletes.

Immutability is what buys the O(1)-memory search the architecture notes promise. Readers never see a half-written segment; they see old files that are still complete or new files that are already complete. The cost shows up in deletes: there is no in-place update. delete_term marks documents dead in a per-segment alive bitset (see src/fastfield/alive_bitset.rs), and the reclaimed space only truly comes back when the IndexMerger rewrites the segment without the tombstoned docs. Every mutating operation gets an incrementing opstamp, which is how the writer serializes concurrent adds, deletes, and commits.

Inside the inverted index

The inverted index is the bread and butter. When a document with the text “Hello, happy tax payer!” arrives, the configured analyzer splits it into tokens — the default tokenizer lowercases and produces [hello, happy, tax, payer] — and the document is registered under terms like text:hello. The term lookup is chained through two structures. First, an FST (finite state transducer) maps Term to a term ordinal in src/termdict/mod.rs. Then a term info store maps that ordinal to a TermInfo, which holds the byte offsets of the posting list inside the .idx file.

Posting lists themselves are blocks of 128 documents: doc ids delta-encoded and bitpacked, term frequencies bitpacked separately, with a variable-int tail for the final partial block. When positions are indexed, a separate .pos file stores token offsets so phrase queries can verify adjacency. The InvertedIndexSerializer writes all of this at seal time; the InvertedIndexReader replays it at query time. Two smaller satellites complete the picture: field norms (src/fieldnorm/writer.rs) store one byte per document — the token count of a field — because the BM25 formula needs it, and fast fields (src/fastfield/writer.rs) are the column-oriented, bitpacked equivalent of Lucene’s doc values, giving you a one-memory-fetch random access by doc id for scoring, filtering, and aggregations.

Scoring and the query machine

Query execution follows a three-trait assembly line. The Query trait is the user-facing tree: QueryParser builds it, BooleanQuery composes sub-queries with Must/Should/MustNot clauses, and TermQuery is the leaf that iterates one term’s posting list. The Weight trait is the per-searcher precomputation — this is where Bm25Weight gathers term and field statistics needed for scoring. Finally the Scorer trait pairs iteration with scoring: it is a cursor over matching doc ids, and boolean queries implement it by merging child cursors.

Underneath everything sits the DocSet trait — a sorted iterator of doc ids, which is what makes intersections and unions cheap. On the consumer side, the Collector trait decides what to do with each (doc, score) pair: TopDocs keeps the top K, while the aggregation collector (src/aggregation/collector.rs) computes histograms, ranges, averages, and stats over fast fields.

From Install to Your First Model

Here is the shortest realistic path to a working index, following the crate’s documented basic search example. Start a fresh project and pull the crate:

cargo add tantivy

Then build a schema, index two documents in memory, and search them. This uses only APIs verified above: Schema::builder, Index::create_in_ram, index.writer, the doc! macro from src/macros.rs, commit, and the reader/searcher pair:

use tantivy::collector::TopDocs;
use tantivy::query::QueryParser;
use tantivy::schema::*;
use tantivy::{doc, Index, TantivyDocument};

fn main() -> tantivy::Result<()> {
    let mut schema_builder = Schema::builder();
    let title = schema_builder.add_text_field("title", TEXT | STORED);
    let body = schema_builder.add_text_field("body", TEXT);
    let schema = schema_builder.build();

    let index = Index::create_in_ram(schema);
    let mut writer = index.writer(50_000_000)?;
    writer.add_document(doc!(
        title => "The Old Man and the Sea",
        body => "He was an old man who fished alone in a skiff.",
    ))?;
    writer.add_document(doc!(
        title => "Of Mice and Men",
        body => "A few miles south of Soledad, the Salinas River drops in close.",
    ))?;
    writer.commit()?;

    let reader = index.reader()?;
    let searcher = reader.searcher();
    let parser = QueryParser::for_index(&index, vec![title, body]);
    for (score, addr) in searcher.search(&parser.parse_query("old man")?, &TopDocs::with_limit(10))? {
        let doc: TantivyDocument = searcher.doc(addr)?;
        println!("{score}: {}", doc.to_json(&schema));
    }
    Ok(())
}

Three habits from the source are worth copying immediately. First, treat commit as the visibility boundary — nothing is searchable before it, as the README’s FAQ states. Second, call reader.reload() if you hold a long-lived reader and want to see later commits; the reload policy can do this automatically on commit. Third, model your delete key explicitly: since Tantivy has no primary id, you pick a field (say, a stored URL or numeric id) and delete_term against it when a document changes.

Storage: Directory trait and the doc store

Everything above touches files through one abstraction, the Directory trait. The crate ships two implementations: MmapDirectory, the default for on-disk indexes whose readers get memory-mapped views (so “loading” an index is essentially just mmapping its files), and RamDirectory for in-memory use. You can implement the trait yourself to put segments anywhere.

The doc store is the row-oriented sibling of the inverted index. StoreWriter compresses each stored document into blocks with LZ4 or Zstd; StoreReader later reconstructs full documents by doc id for result display. The architecture notes are blunt about the cost model: a store fetch walks a skip index and decompresses a block, so it is meant for the top results you will actually render — their rule of thumb is that hitting the store more than a hundred times per query means you are misusing it. Tokenization is pluggable through the TokenizerManager registry, where each named entry produces a TextAnalyzer — a tokenizer chained with filters like lowercasing and stemming.

Honest Limits

Limits stated by the project’s own documentation, not invented here:

• The relevance score is not configurable. Tantivy uses BM25, the same formula family as Lucene and Elasticsearch’s default similarity, per the architecture document.
• Distributed search is explicitly out of scope. The README points you to Quickwit for that layer; the crate itself is a single-node engine library.
• The schema imposes a one-to-one mapping between ingested fields and indexed fields. Wanting one text indexed twice with different analyzers, or indexing a concatenation of several fields, means duplicating or concatenating the fields yourself before feeding them in — the architecture document names this as a current limitation.
• Updates are batch-shaped by design. Adds and deletes are cheap to express but the index is architected around commits and merges; dynamic per-document mutation is not the design center.
• There is no primary key or built-in document identity, so update-by-id is your responsibility (delete term, then re-add).
• The performance numbers floating around — roughly 2x Lucene on their latency benchmark, Wikipedia indexed in about three minutes on the author’s desktop — come from the project’s own README and benchmark page; the README itself warns that your mileage will vary with query and workload.

If this tour landed well, two adjacent reads on this blog: my Elasticsearch architecture tour covers the server-side layer Tantivy deliberately omits, ParadeDB shows what happens when this engine is embedded in Postgres, and for the “search by meaning rather than keyword” side, see FAISS and LanceDB for vector indexes, or Meilisearch for a batteries-included Rust search server.

Enjoyed this post? Never miss out on future posts by following us