Every serious LLM project eventually hits the same wall: you changed one word in a prompt, or swapped one model checkpoint for another, and you have no idea whether the output got better or worse. OpenAI’s answer is evals, the Python framework behind the company’s own model evaluations. It is more than a folder of test cases — it is a small operating system for evaluation: a YAML-driven registry that turns names into Python classes, abstract base classes that define what an eval is, a protocol that abstracts “call the model” behind a single callable, and a recorder that streams every event into an auditable JSONL log. Here we walk that path through the actual code on main, from the moment you type oaieval to the moment a final report lands in a log file.

Two design decisions make this codebase worth studying even if you never run it. First, everything is resolved by name strings: an eval id like test-match maps to a YAML entry, which maps to a class path like evals.elsuite.basic.match:Match, imported and instantiated at runtime — the indirection that lets contributors add evaluations without touching the runner. Second, the framework separates what is evaluated (the Eval) from how the model is reached (a CompletionFn) and how results are stored (a Recorder). If you have read our tour of the community sibling lm-eval-harness, you will notice the opposite emphasis there: a giant task table and metrics library, where evals centralizes the plumbing instead.

The tour below uses two diagrams generated from the repository graph — the first gives the lay of the land, the second drills into one run. Every symbol and line number mentioned was verified against a sparse checkout of openai/evals at main.

The openai/evals overview: the oaieval CLI resolves specs from the Registry, drives the Eval base class over CompletionFn implementations, while every event flows into the Recorder. The elsuite library supplies concrete evals.

Reading the overview from left to right:

• The entry point is the oaieval console script, wired in pyproject.toml to run, L118, which resolves the eval, builds a RunSpec, L75, and drives one evaluation.
• The Registry, L103 lazily loads YAML directories — evals, solvers, completion fns, eval sets — via _load_registry, L287 and resolves names through get_eval, L210.
• Spec objects are plain pydantic dataclasses in evals/base.py: EvalSpec, L51 carries the cls string and args; RunSpec, L75 mints a unique run_id in __post_init__, L85.
• The abstract Eval, L46 class defines the sample loop; its threaded fan-out lives in eval_all_samples, L112.
• Model access is abstracted by the CompletionFn, L23 protocol, implemented for OpenAI endpoints by OpenAICompletionFn, L83 and OpenAIChatCompletionFn, L134.
• Every observation becomes an Event, L44 appended by RecorderBase, L54 and flushed to JSONL by LocalRecorder, L316.
• Concrete evals live under evals/elsuite/: Match, L9 for exact scoring and ModelBasedClassify, L14 for model-graded judgment.

The Registry: YAML In, Python Classes Out

The registry is the framework’s address book. Its constructor takes a list of paths — by default the in-repo evals/registry directory plus ~/.evals, per DEFAULT_PATHS, L30 — and each resource type (evals, solvers, completion fns, eval sets, modelgraded specs) is a subdirectory of YAML files. Loading is deferred: the _evals and sibling collections are functools.cached_property descriptors, so the YAML parses once, on first access, in _evals, L325. Each entry gets its key, its source file’s basename as group, and any class: key renamed to cls — all inside _load_registry, L287, which also rejects duplicate ids and reserved keywords.

Name resolution is the clever part. A registry entry may be a plain string or a dict containing an id, which is just an alias for another entry; _dereference, L156 follows that alias chain until it lands on a real spec, then constructs the typed dataclass. When the CLI asks for an eval that does not exist, the error message is genuinely helpful: difflib.get_close_matches suggests the nearest five ids. Class instantiation happens later via make_object, invoked from get_class, L153, which imports module:Class strings like evals.elsuite.basic.match:Match and injects the spec’s args.

The same machinery serves models. make_completion_fn, L120 recognizes three kinds of names: the literal dummy (a canned DummyCompletionFn, L48 — invaluable for testing without spending tokens), a chat model id like gpt-3.5-turbo detected by is_chat_model, L83, or anything else, looked up as a completion-fn or solver spec and instantiated. Context windows come from a hard-coded table in n_ctx_from_model_name, L37 — a snapshot whose comment honestly dates itself: “Model list last updated 2023-10-24.”

The Detail Map: One Run, End to End

The detail map follows one run: oaieval resolves the eval spec through the registry, the Eval base class fans samples over a thread pool, each sample calls a CompletionFn or a Solver, and match, sampling, and metric events stream into the recorder's JSONL log.

Eval, SolverEval, and the Sample Loop

evals/eval.py defines the contract every evaluation implements. The docstring on Eval, L46 is candid: a subclass overrides two methods — eval_sample, abstract at eval_sample, L77, and run, abstract at run, L86. The constructor takes the resolved completion functions, a registry path, a seed (default 20220722), and the eval’s dotted name, which must have at least a base_eval.split shape.

The fan-out is where the framework earns its keep. _index_samples, L30 shuffles sample indices with a fixed SHUFFLE_SEED of 123 — so runs are comparable even when the caller’s seed changes — and honors a global cap set by set_max_samples, L41 from the CLI’s --max_samples flag. Then eval_all_samples, L112 maps work items over a ThreadPool sized by EVALS_THREADS (default 10), or runs sequentially under EVALS_SEQUENTIAL. Per sample, it binds f"{base}.{split}.{idx}" as a sample id and enters as_default_recorder, L90, a context manager exposing the recorder and sample id through contextvars — which is why deep inside a completion function you can call the module-level record_sampling and it lands in the right place. Samples load via get_jsonl, L120, resolved against the registry’s data directory.

The newer sibling SolverEval, L168 inverts the model: instead of calling a completion function per prompt, it evaluates a Solver, L41 — an agent-like object that receives a TaskState, L24 with a task description, message history, and arbitrary state. SolverEval deep-copies the solver per sample so stateful solvers cannot contaminate each other, and via EVALS_GENTLE_INTERRUPT a Ctrl-C still reports metrics over the samples already finished.

Scoring primitives live in evals/api.py. record_and_check_match, L55 implements the workhorse pattern: prefix-match the sampled text against the expected option(s), require a separator after the prefix if one is defined, and record a match event with expected, picked, and sampled fields. Aggregation happens in evals/metrics.py: get_accuracy, L12 averages the correct flags of match events, while get_bootstrap_accuracy_std, L21 resamples half the events a thousand times to estimate the error bar next to every accuracy number.

CompletionFns, Solvers, and Prompt Formatting

The CompletionFn, L23 protocol is deliberately tiny: a callable taking a prompt (a string or a list of role/content messages) and returning a CompletionResult with get_completions(). The concrete OpenAI implementations wrap the openai client with a hand-rolled retry loop — OPENAI_TIMEOUT_EXCEPTIONS at L19 lists rate limits, connection errors, timeouts, and server errors, and both openai_completion_create_retrying, L27 and openai_chat_completion_create_retrying, L41 retry through it. After each call they record a sampling event including the raw usage, which the CLI later sums into per-run token counts.

Prompt formatting is its own small module. evals/prompt/base.py defines two type aliases — OpenAICreatePrompt, L15 for legacy completions and OpenAICreateChatPrompt, L19 for chat messages — plus two adapter dataclasses: CompletionPrompt, L86 renders a chat log down to User: / Assistant: prefixed text via chat_prompt_to_text_prompt, L22, and ChatCompletionPrompt, L103 lifts a plain string into a single system message. One eval body can thus serve both text-davinci-003-era and chat-era models without branching. Multiple-choice harnesses also get make_abc, L8 for shuffled A) B) C) option strings.

Solvers extend the protocol toward agents. Solver, L41 subclasses CompletionFn (so the registry treats both uniformly) and implements __call__ to deepcopy the task state, run the abstract _solve, then chain any configured PostProcessors — Strip, L5 and friends — over the output, recording each transformation. Composition is handled by NestedSolver, L137, which validates sub-solver specs and caches instances, plus recipes like CoTSolver, L9 and provider solvers such as OpenAISolver, L22. This is the natural seam for tool-using agents: a solver that calls tools shaped like the mcp-python-sdk contract plugs in without changing the eval that grades it.

Model-graded evals — where one model judges another — tie the stack together. ModelBasedClassify, L14 fetches a ModelGradedSpec, L12 from the registry (choice strings, input/output mappings, a prompt template), generates the completions under test, and asks the last completion function in the list to classify them; metaeval mode even grades the grader. Because these evals are pure YAML plus a shared class, the README notes this is currently the only contribution path — custom-code evals are not being accepted.

From Install to Your First Model

The README keeps setup short: pip install -e . from a clone (Python 3.9 minimum), or pip install evals if you only want to run existing evals, with OPENAI_API_KEY exported beforehand. One gotcha: the sample datasets under evals/registry/data use Git-LFS, so a fresh clone shows pointer files until you run git lfs fetch --all; git lfs pull.

Your first run then needs two positional arguments — a completion function name and an eval name — as declared by get_parser, L25:

oaieval gpt-3.5-turbo test-match

That resolves test-match through the registry, instantiates Match, and writes a JSONL log to the default path /tmp/evallogs/{run_id}_{model}_{eval}.jsonl unless --record_path overrides it. Flags worth knowing: --max_samples 20 for a cheap smoke test, --dry_run to swap in the console-only DummyRecorder, --registry_path for your own YAML, and --extra_eval_params for per-eval settings like num_few_shot. The recorder choice is made in build_recorder, L242: local JSONL by default, an HttpRecorder posting batched events under --http-run, or the Snowflake-backed Recorder for OpenAI-internal dashboards. When the run ends, the CLI prints the final report and add_token_usage_to_result, L269 folds summed token counts into it. For suites, oaievalset expands an eval-set YAML into oaieval subprocess commands with checkpointed resume, driven from run, L81.

Honest Limits

Three facts from the README frame your expectations. First, the README’s opening banner points you to configure and run evals “directly in the OpenAI Dashboard” — the project’s center of gravity has moved to a hosted product, and the team is currently not accepting evals with custom code; only model-graded YAML contributions are reviewed. Second, several pillars are frozen in time: the registry’s context-window table is annotated as last updated 2023-10-24, and the completion functions target the openai client of that era — bring your own CompletionFn (the protocol makes that a small class) for newer endpoints or local servers. Third, the Snowflake recorder and its environment variables exist for OpenAI’s internal use; external users realistically use the local JSONL path — the format is just newline-delimited Event dicts with a spec header and a final_report trailer.

None of that diminishes the framework’s value as a blueprint. The registry’s name-to-class indirection, the sample-id-scoped recorder context, and the CompletionFn seam are patterns you can lift into any evaluation stack — including the community harness compared in lm-eval-harness. And the lineage runs deep: many registry datasets still reference completions from the gpt-2 era, when “evaluation” meant perplexity and exact match.

Enjoyed this post? Never miss out on future posts by following us