Every benchmark leaderboard you have seen — MMLU, HellaSwag, GSM8K — is computed by software that has to make hundreds of small, boring decisions correctly: how the prompt is assembled, how many few-shot examples are drawn and from which split, how the model’s answer is extracted from raw generated text, and how per-sample numbers collapse into one headline figure. For open models, that software is often the LM Evaluation Harness from EleutherAI — the framework behind the Open LLM Leaderboard lineage and the de facto reference for reproducible LLM scoring.

The library has a reputation for being “just YAML”, and the YAML surface is real — the lm_eval/tasks directory on the main branch contains roughly fourteen thousand task configuration files. But underneath that declarative skin sits a compact, elegant runtime: an evaluator core that walks every task, a request abstraction that turns documents into four kinds of scoring jobs, a set of model adapters that all answer the same two verbs, and a metric/filter layer that converts free-form text into numbers. This post walks that runtime from the command line down to the token logits, with every symbol linked to the actual source at the commit we read.

We will follow one thread the whole way: what exactly happens between lm_eval --model hf --tasks mmlu --limit 100 and a JSON file full of accuracies. If you evaluate models for a living — or you are building your own harness and want to steal the good ideas — this is the map.

Overview of the harness on the main branch: the CLI funnels into simple_evaluate, which loads YAML tasks through TaskManager, turns documents into Instance requests via ConfigurableTask, dispatches them to a backend such as HFLM or VLLM, and scores the filtered responses with the metric registry.

Reading the overview from left to right:

• cli_evaluate, L5 is the console entry point; it builds a HarnessCLI, L10 parser whose run subcommand (Run, L18) translates flags into a library call.
• simple_evaluate, L55 is the one-call orchestrator: it fixes all four random seeds, resolves the model string through the registry, wraps the model in CachingLM, L275 if you asked for response caching, and overrides per-task settings like num_fewshot.
• TaskManager, L37 maintains a cached TaskIndex, L36 over the YAML forest and its load method (L179) resolves names, tags, and inline dicts into live task objects via TaskFactory, L25.
• ConfigurableTask, L618 is the workhorse: one Python class that interprets a TaskConfig, L82 dataclass — dataset path, splits, doc_to_text, metric_list, filter_list — and implements the whole lifecycle declared in the abstract Task, L64.
• ContextSampler, L17 draws the few-shot examples for fewshot_context, L933, with FirstNSampler, L104 and BalancedSampler, L117 as deterministic and per-label alternatives.
• evaluate, L429 is the request loop: after build_all_requests, L268 fills each task with Instance, L11 objects, it groups them by request type and executes resps = getattr(lm, reqtype)(cloned_reqs) at L600 — the request type literally is a method name on the model.
• On the way out, RegexFilter, L16 extracts answers from generated text, and process_results, L1455 computes per-sample metrics that the aggregation registry — for instance mean, L35 — collapses into the final score.

From Install to Your First Model

The package installs as lm_eval (the CLI alias is the same name), and the fastest smoke test uses a tiny checkpoint:

pip install lm-eval
lm_eval --model hf --model_args pretrained=EleutherAI/pythia-70m \
  --tasks hellaswag --limit 100 --batch_size 8

Three things happen that are worth understanding before you scale up. First, the model string hf is not a class import; it is a registry key. The module lm_eval/models/__init__.py registers every known adapter alias lazily in _register_all_models, L66, and simple_evaluate resolves it with get_model, L613 — which is also how third-party packages plug in backends through entry points. Second, --model_args pretrained=... is parsed into a dict and passed to the adapter constructor, so the flags you pass are just keyword arguments of HFLM.__init__. Third, the task name hellaswag goes through the task index, which maps the name to a YAML file; the YAML in turn names the datasets split and the prompt template. Nothing about HellaSwag is hard-coded — it is all configuration, which is why the community has contributed thousands of tasks.

The same call also demonstrates the reproducibility contract: simple_evaluate sets separate seeds for Python’s random, numpy, torch, and the few-shot sampler, then records all of them, plus the git hash and environment info, into the results object (EvalResults, L19). If two runs disagree, the JSON tells you what differed.

How a Task Becomes a Request

The heart of the design is a four-value enum of request types, declared in lm_eval/api/instance.py: loglikelihood, loglikelihood_rolling, generate_until, and multiple_choice. Every benchmark reduces to one of these verbs: a classification benchmark is multiple_choice (argmax over choice likelihoods), open-ended generation is generate_until, and perplexity benchmarks use the rolling variant.

ConfigurableTask.construct_requests (L1362) shows how far one method can stretch. For multiple_choice it builds one loglikelihood request per choice, gluing the target delimiter onto each continuation; if the task requested acc_mutual_info, it additionally builds unconditional (empty context, choice) pairs so the score can be normalized by the choice’s unconditional probability. For generate_until it attaches a deep copy of the task’s generation_kwargs so per-task sampling settings cannot bleed into each other. The resulting Instance dataclasses — request type, source document, arguments, and later the responses — are cached as a whole by build_all_requests through load_from_cache, L53, keyed by task, shot count, rank, and chat-template settings.

The few-shot context itself is assembled in fewshot_context, L933, which builds a message list: optional system instruction plus the task description, then num_fewshot examples drawn by the sampler (which removes the current evaluation document from its own draw), then the evaluation document without its answer. If apply_chat_template is set, the same message list is rendered through the tokenizer’s chat template — with fewshot_as_multiturn deciding whether prior examples become separate user/assistant turns or one concatenated user message.

The detailed view adds the internals: CLI subcommands, the evaluator's helpers and result schema, the task index/factory/index machinery, the samplers, the LM → TemplateLM → HFLM/VLLM/TemplateAPI adapter hierarchy, the filter ensemble, and the logging layer.

Backends: Swap Without Rewrites

All model adapters implement the same narrow interface declared in LM, L25: answer loglikelihood and generate_until batches, expose rank/world_size for distributed runs. A shared base class, TemplateLM, L356, owns the tokenization boilerplate — most importantly _encode_pair, L393, which encodes context and continuation, moves trailing whitespace into the continuation to preserve word-boundary tokenization, and splits the whole-sequence encoding to get exact continuation token IDs. Get this function wrong and every likelihood benchmark silently changes meaning; the harness gets it right once, in one place.

The default adapter, HFLM, L62, wraps the Transformers library directly. Its scoring path batches requests, runs one forward pass, and slices the continuation logits; its generation path _model_generate, L1153 defaults the temperature to 0.0 and, unless you explicitly set do_sample, strips the temperature and runs greedy decoding with custom stop-sequence criteria. Determinism is the default; sampling is opt-in.

For throughput, the VLLM, L59 adapter swaps in vLLM as the engine. The trick: scoring does not change conceptually — _model_generate, L388 issues a request with SamplingParams(temperature=0, prompt_logprobs=1, max_tokens=1, detokenize=False) — one forward pass per prompt with log probabilities enabled — which reproduces HFLM’s likelihood numbers at vLLM’s batched speed. The adapter also handles tensor-parallel serving, Ray-based data-parallel replication, and optional LoRA adapters. If your model is served behind an HTTP endpoint instead, TemplateAPI, L105 speaks to any JSON API — including TGI deployments and OpenAI-compatible servers — with a pluggable payload template, retry logic in model_call, L453, and an async path with semaphore-based concurrency.

Scoring: Filters, Metrics, Aggregation

Generation tasks produce prose; metrics want values, and the harness splits that problem in two: filters clean the raw text, metrics score the cleaned text. A task’s YAML lists any number of filter chains in filter_list, and ConfigurableTask builds each one at init time via build_filter_ensemble, L11, which looks components up in the filter registry and wraps them in a FilterEnsemble, L34. The most-used component is RegexFilter, L16 — its default pattern, #### (\-?[0-9\.\,]+), is a giveaway that it was born for GSM8K-style answers — and when nothing matches it returns the fallback [invalid] rather than crashing. MultiChoiceRegexFilter, L123 instead picks the answer choice that appears in the response. Each filter chain produces its own filtered_resps bucket, so one run can report exact-match under two extraction rules simultaneously.

Metrics themselves are registered, not imported: register_metric, L707 attaches to each metric name its aggregation function, output types, and direction (higher_is_better), and task YAML just names the metrics. process_results computes raw per-sample values — for multiple_choice that means the argmax over choice log-likelihoods, plus byte- and character-length-normalized variants for the well-known acc_norm — and the evaluator collects them into per-metric lists. At the end, _process_results, L375 applies the registered aggregation (exact_match_hf_evaluate, L234 under a mean, perplexity as exp(-mean(loglikelihood))), bootstraps standard errors, and folds task scores upward through ConfigurableGroup, L346 nodes so that an umbrella benchmark like mmlu aggregates from its subtasks with the weighting its YAML specifies. Results and samples stream out through the tracker (EvaluationTracker, L123) and optionally to WandbLogger, L24.

Distributed correctness gets real attention: when world_size > 1, evaluate pads each rank’s request list so every rank performs an equal number of forward passes, runs a barrier after each request type, and gathers metrics and logged samples to rank 0 before aggregation.

Honest Limits

The YAML surface is a strength and a liability. Fourteen thousand configuration files are maintained by hand, and their sheer number makes review hard; task definitions drift, near-identical benchmarks get duplicated under different names, and prompt-format choices that matter enormously for scores stay invisible unless you read the file.

Extraction brittleness is the second caveat. RegexFilter’s fallback [invalid] keeps a run alive when the model rambles, but every such sample silently becomes a wrong answer for exact-match metrics. A model that answers correctly but in an unexpected format gets punished for formatting, not reasoning — and with chat-template variants this makes cross-paper score comparisons genuinely unsafe unless the template settings match.

Third, the multiple-choice normalization heuristics (acc_norm dividing by character length, the byte-length variant) are exactly that — heuristics with known failure modes on short-choice tasks. And the distributed padding logic carries its own in-code caveat: it pads per request type and does not account for tasks that mix request types within one document. Nothing here invalidates the numbers, but these are decisions the harness makes on your behalf, and the only defense is knowing they exist.

Finally, defaults encode opinions worth re-checking: bootstrap_iters=100000 for stderr costs time on big runs, and predict_only bypasses metrics entirely. And no harness can save you from contamination between a model’s pretraining data and a public benchmark — that is a property of the corpus, not the tool. Use it as a measurement instrument, read the task YAML for anything you publish, and treat cross-model comparisons as valid only when the shot counts, templates, and filter chains match. Within those rules, it remains the most trustworthy open scoring tool the ecosystem has — and reading its source, as we did here, is the fastest way to internalize what a benchmark score actually measures.

Enjoyed this post? Never miss out on future posts by following us