Every benchmark number you have seen attached to an open model - MMLU, HellaSwag, GSM8K, ARC, TruthfulQA - has a decent chance of having been produced by this repository. EleutherAI’s Language Model Evaluation Harness is the framework behind the Open LLM Leaderboard and hundreds of papers, and its power comes from an architecture that treats evaluation as data: every task is a YAML file describing prompts, document formats, and metrics, and the framework’s job is to turn any language model into a scoring function against those files. Release 0.4.14 under the MIT license ships with 221 task directories under lm_eval/tasks, each holding configs that expand into hundreds of benchmark subtasks and variants.

What makes the source worth reading is that it is a masterclass in interface design under constraint. The whole system communicates through two verbs - loglikelihood and generate_until - so anything that can answer those two requests becomes an evaluable model, from a Hugging Face checkpoint to a vLLM engine to a commercial API to a llama.cpp GGUF file. Around that spine sits a task system powerful enough to encode fewshot selection, answer extraction, and metric aggregation as declarative config, plus a plugin registry that keeps 30 model backends from tangling the core. As always in this series, what follows is an educational tour of published source code.

Overview: the CLI drives the evaluator, which pulls tasks from the manager and requests from the model abstraction; answers flow through filters into aggregation and result logging.

Reading the overview from left to right:

• lm_eval/_cli/run.py is the 518-line entry that parses CLI flags into a simple_evaluate call.
• lm_eval/evaluator.py holds the two public functions: simple_evaluate at line 55 and the core evaluate loop at line 429 of its 714 lines.
• lm_eval/tasks/manager.py defines TaskManager at line 37, the indexer that turns task names into config-loaded classes.
• lm_eval/api/task.py is the 1808-line heart: Task at line 64, ConfigurableTask at line 618.
• lm_eval/api/model.py declares the LM interface at line 25 with TemplateLM at line 356 and CachingLM at line 275.
• lm_eval/models/huggingface.py is the reference backend, HFLM at line 62 of 1782 lines.
• lm_eval/api/filter.py pairs the Filter contract at line 8 with FilterEnsemble at line 34, fed by regex extractors in filters/extraction.py.
• lm_eval/loggers/evaluation_tracker.py persists everything through EvaluationTracker at line 123.

Why You Need This

If you have followed the fine-tuning tours here - LLaMA-Factory, axolotl, unsloth - you have watched the training side of the loop. The harness is the other half: the tool that tells you whether a model got better. Reading it teaches you what benchmark numbers actually measure, because every step that could silently change a score is explicit: which prompt template was used, how many fewshot examples were sampled and by what rule, how the raw generation was cut down to an answer, and which aggregation produced the table cell. Reproducibility is not a slogan here; it is enforced by config files checked into the repo.

The backend abstraction is the lesson most applicable elsewhere. Anything implementing loglikelihood and generate_until plugs in - and the catalog shows the range: HFLM for transformers checkpoints with quantization support, VLLM for batched inference throughput, TemplateAPI for OpenAI-compatible endpoints, Megatron for very large models, plus GGUF, ONNX, and Neuron variants. The same interface idea powers ctranslate2 and onnxruntime in serving, and seeing it applied to evaluation makes a strong argument for tiny, honest interfaces over framework lock-in.

The task system deserves study if you ever design datasets or leaderboards. ConfigurableTask turns a YAML file into prompt construction, target extraction, and metric selection, while the filter ensemble composes regex extraction, whitespace trimming, and majority-vote selection into the answer stage. The metric library covers everything from exact match and F1 to BLEU, chrF, perplexity, and bits-per-byte, with aggregation declared per task so nothing is hard-coded. Once you have seen how much behavior fits in declarative config, you will never hard-code an eval script again.

Detailed view: CLI entry, the evaluation core, the task API with metrics and samplers, the backend family registered against the LM interface, task definition loading, filters, and logging with caching.

The main flow starts at simple_evaluate at line 55 of evaluator.py, which resolves the model through get_model at line 613 of api/registry.py, builds the task list through a TaskManager constructed in tasks/manager.py, applies decontamination and limits, and then hands everything to evaluate at line 429 - the loop that batches requests per task, sorts them across tasks for throughput, calls the model, and routes outputs through filters and aggregation. The supporting math lands in evaluator_utils.py: get_sample_size at line 49, the aggregation accumulator at line 99, group-aware rollups through aggregate_groups at line 301, and result shaping in _process_results at line 375 of its 540 lines.

The registry is the hinge of the whole design. Registry at line 270 of api/registry.py is a generic plugin container fed by load_plugins at line 189 and import_plugins at line 246; register_model at line 587 lets any backend class join the catalog by name, DEFAULT_METRIC_REGISTRY at line 571 maps task output types to default metrics, and get_model at line 613 is the single factory call the evaluator makes. This is why the models directory can hold thirty backends - anthropic, openai_completions, textsynth, sglang, trtllm, neuron, watsonx - without the core importing any of them.

The breadth of that directory repays a skim. HFMultimodalLM at line 30 of models/hf_vlms.py extends HFLM across its 750 lines for vision-language checkpoints; MambaLMWrapper at line 10 of models/mamba_lm.py shows a state-space model joining the club through the same base class in just 164 lines; GGUFLM at line 24 of models/gguf.py implements the LM interface directly to talk to llama.cpp; and TRTLLM at line 37 of models/trtllm_causallms.py covers TensorRT-LLM deployments in 631 lines. Even audio generation gets a slot through hf_audiolm.py, and the thinness of the smaller wrappers is the point - the easier a backend is to write, the better the interface is doing its job.

Every backend implements the same small interface from api/model.py: LM at line 25 declares loglikelihood, loglikelihood_rolling, and generate_until; TemplateLM at line 356 implements the shared tokenization glue for models that expose token access; CachingLM at line 275 wraps any backend with a sqlite cache keyed by hash_args at line 255, so repeated runs replay identical requests for free. The reference implementation, HFLM at line 62 of models/huggingface.py, handles device placement, batching, PEFT adapters, and quantized checkpoints across its 1782 lines; VLLM at line 59 of models/vllm_causallms.py swaps throughput into the same contract; and TemplateAPI at line 105 of models/api_models.py does the same for remote endpoints, with shared helpers like handle_stop_sequences at line 641 of models/utils.py keeping behavior aligned.

The task layer is where YAML becomes evaluation. TaskManager at line 37 indexes every task config, tasks/_yaml_loader.py loads them with include-resolution and safe function imports through load_yaml at line 164, and tasks/_factory.py builds concrete tasks through TaskFactory at line 25. The resulting ConfigurableTask at line 618 of api/task.py is an engine of over a thousand lines that constructs fewshot contexts through ContextSampler at line 17 of api/samplers.py (with FirstNSampler, BalancedSampler at line 117, and ManualSampler variants), builds Instance requests from api/instance.py, scores outputs with the metric functions in api/metrics.py - mean at line 35, perplexity at line 47, f1_score at line 62, bleu at line 102 - and specializes into MultipleChoiceTask at line 1688 and PerplexityTask at line 1735 for the two classic evaluation shapes. Groups compose tasks hierarchically through Group at line 34 of api/group.py and ConfigurableGroup at line 346.

Answer extraction gets its own layer because raw generations rarely match gold strings directly. Filter at line 8 of api/filter.py is the contract and FilterEnsemble at line 34 chains several; filters/extraction.py provides RegexFilter at line 16, a POS-aware filter at line 66, WhitespaceFilter at line 110, and MultiChoiceRegexFilter at line 123 for multiple-choice formats, while filters/selection.py implements majority voting over sampled generations. Results finally flow through the logging stack: GeneralConfigTracker at line 38 and EvaluationTracker at line 123 of loggers/evaluation_tracker.py write samples, configs, and tables to disk or the Hub, loggers/wandb_logger.py streams the same to experiment dashboards, and the 88 lines of caching/cache.py back the request cache that makes reruns nearly instant. Even discovery is a first-class feature - lm_eval/_cli/ls.py powers lm-eval ls tasks and ls models against the same index the runner consumes.

From Install to a Benchmark Table

Install with the backends you need, then evaluate:

pip install "lm_eval[hf,vllm]"
lm_eval --model hf \
    --model_args pretrained=EleutherAI/pythia-1.4b \
    --tasks hellaswag,arc_easy \
    --device cuda:0 --batch_size 8

The same run from Python:

import lm_eval

results = lm_eval.simple_evaluate(
    model="hf",
    model_args="pretrained=EleutherAI/pythia-1.4b",
    tasks=["hellaswag", "arc_easy"],
    num_fewshot=0,
)
print(results["results"])

Both paths return per-task metric tables plus samples you can inspect, and the tracker writes them to JSON for later comparison.

Honest limits: benchmark scores are proxies, and the harness makes prompt and extraction choices visible but cannot make them neutral; a few backend integrations lag the latest upstream releases; and running 221 task groups naively costs real money and compute, so sane usage starts with limit and a hand-picked task list. The value of the source is that none of these limits are hidden - every choice is a line you can read.

Enjoyed this post? Never miss out on future posts by following us