Marqo is an open-source vector search engine with a simple pitch: you hand it documents — text, images, or mixed content — and it generates embeddings for you, stores them, and lets you query them in natural language through a single REST call. Instead of bolting an embedding model onto a separate vector database yourself, Marqo treats “which model embeds my data” as an index setting and handles the whole flow, from media download and preprocessing to ANN ranking, inside one system. This tour walks the actual source of marqo-ai/marqo at branch mainline (version 2.26.2, per components/common/src/marqo_common/version.py), and every symbol below is linked to the exact line I read.

The deployment shape is the first thing that surprised me. Marqo is not a single Python process: the repository is split into four Python components under components/ — the main API server (marqo), an inference orchestrator (inference_orchestrator), a model management container (model_management), and shared code (common). The root compose.yaml wires those together with NVIDIA Triton Inference Server and expects a Vespa node reachable on the host. Ports tell the story: the API listens on 8882 (uvicorn.run(app, host="localhost", port=8882) in components/marqo/src/marqo/tensor_search/api.py), model management on 8883, the orchestrator on 8884, and Triton on 8000/8001/8002. Embeddings never happen in the API process — they are shipped over HTTP to the orchestrator, which drives Triton over gRPC.

Underneath it all sits Vespa as the storage engine. The VespaClient in components/marqo/src/marqo/vespa/vespa_client.py is the only thing that talks to it: YQL queries in, document feeds out, application packages deployed when schemas change. Marqo’s “index” is really a generated Vespa application — schema files rendered from Jinja templates — which is why index creation is a heavyweight operation in this codebase. If you have followed my earlier posts on USEARCH or pgvector, you will notice the opposite design philosophy: those tools are minimal storage engines you drive yourself, while Marqo owns the full stack from model to index.

Marqo overview: the FastAPI server (left) wires core services through Config; documents and queries flow through the Vespa store (bottom), while every embedding request detours to the inference orchestrator and Triton (right). Click any node to open the source on GitHub.

Reading the overview from left to right:

• app = FastAPI(...), L111 — the API entrypoint that mounts every route: create index, add documents, search, embed, recommend, model management, and health.
• generate_config(), L51 — runs once at import time and builds the three remote clients: VespaClient, InferenceClient, and ModelManagerClient, then wraps them in a Config.
• Config, L22 — the wiring hub; its constructor instantiates IndexManagement, Monitoring, Document, Recommender, Embed, and Typeahead, each receiving exactly the dependencies it needs.
• IndexManagement, L29 — owns index lifecycle: create_index at L128 generates and deploys a Vespa application package; get_index at L588 reads settings back.
• Document.add_documents, L44 — the bulk add entrypoint; it picks one of three index-specific handlers and lets that handler vectorise and feed documents.
• tensor_search.search, L324 — the root search method; validates limits, fetches cached index settings, then dispatches by searchMethod (TENSOR by default).
• InferenceClient.vectorise, L31 — every embedding, whether from indexing or querying, leaves the API process through this HTTP call to the orchestrator’s POST /vectorise handler at L88 of components/inference_orchestrator/src/inference_orchestrator/main.py.
• VespaClient, L40 — pooled HTTP client with query at L279 and feed_batch_multithreaded at L421; also handles Vespa application deployment at L95.
• HybridSearch, L64 — runs the tensor and lexical legs together and merges their result sets when searchMethod=HYBRID.

The Detail View

The detail diagram zooms into the six functional areas the overview hides: the API layer, the Config-wired core services, the search core, the Vespa store adapters, the Marqo-side inference client, and the two sidecar services.

Detail view: from generate_config down to Triton gRPC calls. Three Vespa index adapters — structured, semi-structured, and unstructured — show how one backend serves three index flavours.

Two things in this view deserve attention. First, MarqoCustomRoute in components/marqo/src/marqo/api/route.py (L9) is a custom FastAPI route class that logs full stack traces before re-raising, and api.py registers a large ordered exception-mapping table (L137–180) that converts core, Vespa, and inference exceptions into consistent HTTP responses. Second, CachingInference (L14 in caching_inference.py) is a decorator around InferenceClient: when MARQO_API_INFERENCE_CACHE_SIZE is greater than zero, repeated content skips the network round trip entirely, backed by pluggable LRU/LFU caches.

From Install to Your First Model

The quickest smoke test, straight from components/marqo/src/marqo/README.md, is a single container: docker run --name marqo -p 8882:8882 marqoai/marqo:latest. That image bundles enough of the stack to serve requests on port 8882, and http://localhost:8882/docs exposes the live Swagger UI generated from the FastAPI app (the README also shows how to pull the raw openapi.json).

For a real run you want the full component stack from the repo. Start a Vespa container (docker run --detach --name vespa --publish 8080:8080 --publish 19071:19071 vespaengine/vespa:latest), deploy Marqo’s local application package to it with the prepareandactivate curl documented in the developer guide, and then bring up the Python side with docker compose -f compose.yaml up, which starts Triton, the API, model management, and the orchestrator with the right environment variables — VESPA_CONFIG_URL, VESPA_QUERY_URL, VESPA_DOCUMENT_URL all point at host.docker.internal, and MARQO_REMOTE_INFERENCE_URL points at the orchestrator. Developers who want breakpoints can skip the API container entirely and run uvicorn api:app --host 0.0.0.0 --port 8882 from src/marqo/tensor_search, as the README’s Option B describes.

Your first index is one POST against a verified route (@app.post("/indexes/{index_name}"), L315 of api.py):

curl -X POST http://localhost:8882/indexes/my-first-index \
  -H 'Content-Type: application/json' \
  -d '{"tensorFields": ["description"]}'

IndexSettings in components/marqo/src/marqo/tensor_search/models/index_settings.py defaults to a semi-structured index with model hf/e5-base-v2 (L27), so the minimal body is enough — Marqo will download and load that HuggingFace model on first use. Adding documents hits the /documents route (L550), whose AddDocsBodyParams accepts a documents list of plain dicts (components/marqo/src/marqo/api/models/add_docs_objects.py):

curl -X POST http://localhost:8882/indexes/my-first-index/documents \
  -H 'Content-Type: application/json' \
  -d '{"documents": [{"_id": "post-1", "description": "A short tutorial about vector search"}]}'

Then search — the SearchQuery model (models/api_models.py L42) defaults to searchMethod: TENSOR and limit: 10:

curl -X POST http://localhost:8882/indexes/my-first-index/search \
  -H 'Content-Type: application/json' \
  -d '{"q": "embeddings tutorial"}'

The first search may feel slow while the model loads into Triton, but after that, query embeddings are cached and the ANN search itself is Vespa’s job.

How a Search Query Travels

tensor_search.search (L324) is the root of the tree. It validates limits (MARQO_MAX_SEARCH_LIMIT, MARQO_MAX_SEARCH_OFFSET, and friends), then pulls the index definition through index_meta_cache.get_index (L44 of index_meta_cache.py), an in-process cache with a background refresh thread so hot searches do not re-read settings from Vespa. Dispatch then branches on searchMethod: _vector_text_search (L1124) embeds the query text through the inference client and asks Vespa for nearest neighbours; _lexical_search (L558) builds a keyword query instead; and HybridSearch.search (L65) executes both and merges. The Marqo-to-Vespa translation happens in the index adapters — StructuredVespaIndex builds tensor, lexical, and hybrid YQL queries (to_vespa_tensor_query at L419 of structured_vespa_index.py) — and VespaClient.query (L279) executes them. Highlights come back out of the response in _extract_highlights (L1030).

There is also a query-free recommendation path: the /recommend endpoint (L492 of api.py) calls Recommender.recommend (L199 of recommender.py), which fetches stored vectors for seed document IDs, interpolates them, and searches — the docstring is explicit that no inference happens during this process.

The Bulk Add Path

The ingestion flow is the more interesting half of the system. After the API parses AddDocsBodyParams, Document.add_documents (L44) reads the index definition and instantiates exactly one of three handlers: StructuredAddDocumentsHandler, SemiStructuredAddDocumentsHandler, or UnstructuredAddDocumentsHandler (L57–66 of document.py). They share a base class, AddDocumentsHandler (L107 of core/vespa_index/add_documents_handler.py), whose add_documents (L127) runs a fixed sequence: validate every document, infer modality per tensor field, convert the batch into Vespa documents (_convert_to_vespa_docs, L246), vectorise tensor fields through the inference client (_vectorise_tensor_fields, L283), and finally feed everything with VespaClient.feed_batch_multithreaded (L421). Failures are collected per document rather than failing the whole batch, and the response reports errors indexed by original list position.

Media handling is delegated to the orchestrator side: when treatUrlsAndPointersAsMedia is set, the API sends URLs and the orchestrator’s media_download_and_preprocess service downloads images or video and chunks them before embedding. Text fields go through configurable chunking too — IndexSettings defaults to sentence-based splits (index_settings.py L32–36). Partial updates use a separate PATCH route (L572 of api.py) and VespaClient.update_documents_batch (L601), so re-embedding is avoided when only non-tensor fields change.

Index Types and Schema Generation

Creating an index is where Marqo shows its Vespa inheritance. IndexManagement.create_index (L128) renders a VespaApplicationPackage (class at L593 of vespa_application_package.py) — schemas generated from .sd.jinja2 templates plus the services definition — and deploys it through VespaClient.deploy_application (L95). For clustered setups, a Zookeeper-backed distributed lock (_vespa_deployment_lock, L709) serialises deployments, and Config._connect_to_zookeeper (L69 of config.py) degrades gracefully to a warning if Zookeeper is unreachable. The three index types trade flexibility for schema stability: structured indexes predeclare every field, semi-structured (the default, L20 of index_settings.py) adapts the schema as new field counts grow, and unstructured indexes keep the old dynamic-mapping behaviour. ANN tuning is exposed up front: HNSW with efConstruction=512, m=16, and prenormalised angular distance (L49–55).

Honest Limits

Start with the headline: the project README states that Marqo’s open-source project is deprecated and will no longer receive updates, pointing commercial users to the managed platform at marqo.ai. The mainline branch remains a fascinating read and a working system, but plan for maintenance being on you. The second limit is weight: a full stack is five containers plus Vespa’s JVM, and index creation triggers real deployments — if you want an embedded engine that lives inside your process, my earlier notes on USEARCH or pgvector cover that end of the spectrum. Third, inference is always remote in this architecture: the API has no in-process embedding fallback, so a broken orchestrator or Triton takes embeddings down with it, and model loading is governed by MARQO_MODELS_TO_PRELOAD and the model management container. Finally, as wired in compose.yaml, the HTTP surface has no authentication layer — only telemetry middleware is registered (L118–119 of api.py) — so it is meant to sit inside a trusted network. Model coverage is nonetheless a genuine strength: the registry resolves HuggingFace transformers checkpoints such as the default hf/e5-base-v2, CLIP-family vision-language models through OpenCLIP, and ONNX-friendly setups behind Triton.

Enjoyed this post? Never miss out on future posts by following us