Everyone accumulates a private mountain of notes, PDFs, and half-remembered web pages. Khoj is the AGPL-3.0 personal AI that turns that mountain into something you can actually talk to: a self-hostable second brain that ingests your markdown, PDFs, org-mode files, Word docs, Notion pages, and GitHub repos, then lets you chat with any of the major LLMs — local or hosted — about all of it at once. It scales from an on-device assistant to a cloud deployment, reaches you through a web app plus Obsidian, Emacs, desktop, and Android clients, and layers on automations, custom agents, image generation, voice, and even computer-use operators that can drive a browser for you. The README’s benchmark claims are modestly understated; the feature list is not.

Architecturally it is a Django monolith done right: one models file holds every table, format-specific ingestion processors funnel into a shared entry builder, a compact search engine blends embeddings with classic scoring, and a very large helpers module orchestrates whatever tools the conversation needs. Born as an Emacs search package and grown into a full product with a cloud offering, it wears its evolution openly in the source tree. As always in this series, what follows is an educational tour of published source code.

Khoj at a glance: a Django server with one shared data model on top, ingestion processors feeding a semantic search core, a conversation brain that grounds answers in your documents, and clients everywhere.

Reading the overview from left to right:

• The Django entry — the ASGI application that runs migrations and loads ML models in a background thread at startup.
• The configurator — mounts every router, middleware, and scheduled job when the server boots.
• The data models — all Django models in one file: users, entries, embeddings, agents, automations.
• The content processors — format adapters sharing one entry builder that deduplicates, chunks, and diff-updates.
• The embedder — local or hosted embedding models with vectors cached as compressed binary blobs.
• The semantic search — blended embedding and tf-idf scoring with optional cross-encoder reranking.
• The chat helpers — the orchestration core choosing tools, running research, and streaming answers.
• The deep research loop — an agentic planner that searches, reads, and iterates before responding.

Why You Need This

The first reason is the ingestion design, which solves the boring half of every personal-knowledge project: keeping an index fresh without hammering your sources. Each format adapter in processor/content converts its source into plain text, and the shared entry builder hashes every entry, skips duplicates, and only updates what actually changed when you resync. That incremental diffing is why Khoj can sit on top of a live Notion workspace or a Git repository without re-reading the world on every cycle, and why syncing stays fast even as the corpus grows. The embedder then caches vectors as compressed binary blobs, so restarts are cheap and local-first stays genuinely local.

The second reason is that search here is a masterclass in pragmatism. The semantic search module is barely two hundred lines yet blends vector similarity with tf-idf scores and optional cross-encoder reranking, deliberately avoiding a heavyweight vector database dependency. The date filter parses human phrases like “from: last month” out of natural queries, with sibling filters for file types and exact words. If you have ever wanted to understand how far good engineering, rather than expensive infrastructure, can take retrieval quality, these three files are the case study.

The third reason is the conversation stack’s honesty about providers. The chat helpers — the largest file in the repo — route a single conversation across OpenAI, Anthropic, Google, and offline models, normalizing tool calls and streaming into one coherent flow. When a question deserves more than one pass, control escalates to the deep research loop, which plans sub-queries, reads sources with visible progress, and iterates until it can answer with citations. A prompt library in processor/conversation/prompts.py keeps every instruction in one auditable place, so you can see exactly what the system tells your models.

The detail view: server and auth at the top, ingestion adapters converging on the entry builder, the search core with its filters, the conversation brain with provider adapters, and the tool belt beside it.

The detail diagram shows a clean funnel. Everything you feed Khoj — markdown, PDF, org-mode, Notion, GitHub — narrows into the entry builder and lands in the single Django models file, which is the whole database in one readable file. The search group reads from those tables, with the three filters composing inline in any query. On the right, the conversation brain fans out: helpers pick tools per turn, the provider adapters speak each vendor’s dialect, and the tool belt ranges from online search to MCP integration, sandboxed code execution, and computer-use operators that drive browser and desktop environments through provider-specific agents. Custom agents in api_agents.py customize persona, knowledge, and model per agent, automations schedule recurring jobs like personal newsletters, and image plus speech generation round out the senses.

The clients group is a reminder of the project’s Emacs roots. The web app is the primary interface today, but the Obsidian plugin and Emacs package query the same API, and the desktop client simply wraps the web experience in an Electron shell. One server, many doors — no per-client logic to drift out of sync, and every interface inherits improvements to search and chat the moment the server updates, which is exactly how a multi-client product should behave.

From Install to a Chat With Your Notes

The official path is Docker: grab the docker-compose.yml from the repository, set your administrator account and one provider key, and docker compose up brings up the server on localhost:42110, where the web app walks you through connecting your first documents. For development, the repo ships a devcontainer and a setup script under scripts that provisions Postgres and all Python dependencies. Once running, upload or connect a folder of notes, wait for the indexing jobs to finish, then just ask a question — Khoj decides whether to answer from your knowledge, search the web, or escalate to deep research, and streams its reasoning as it goes so you are never staring at a silent spinner. Point the Obsidian plugin or the Emacs package at the same server and your second brain follows you into the editor, searchable without breaking your writing flow.

Honest limits: Khoj is AGPL-3.0, which matters if you embed it in a hosted product; the helpers module’s size makes the chat flow harder to follow than the rest of the codebase; self-hosting assumes comfort with Docker and Postgres; and the richest features — automations, computer-use operators, voice — lean on hosted model APIs rather than fully offline setups. The Android client also lags the web experience. Even so, for a readable, end-to-end example of building a personal AI over your own corpus, it remains one of the best codebases to study.

Enjoyed this post? Never miss out on future posts by following us