Skip to content

Evaluation Overview

This document explains how Marin evaluates models and where to find runnable workflows.

For step-by-step usage, start with:

Evaluation modes

Marin supports three evaluation paths:

  • In-loop training evals: train_lm runs periodic multiple-choice evaluations through Levanter's lm-evaluation-harness integration and logs to W&B when an EvalSuite is provided.
  • Post-hoc evals: the shared launcher or composable EvalGroups evaluate multiple-choice and generation tasks with the Evalchemy fork.
  • Harbor tasks: the shared launcher runs containerized agent benchmarks and registry datasets through Marin's Harbor integration.

Post-hoc evaluation (evalchemy over a served endpoint)

A post-hoc eval is decoupled from the model backend by an OpenAI-compatible URL. Each EvalGroup is run inside one remote_inference context. The generic group runner passes its RunningModel to run_evalchemy, then inference is torn down. Multiple-choice tasks use the served backend's logprob API, so they run the same way as generation — no separate JAX-logprob backend.

The lifecycle and the evaluator are separate APIs:

with remote_inference(model, engine, iris) as session:
    run_evalchemy(session.model, eval_config, output_path, env_vars=env)

remote_inference owns startup, liveness, Iris link registration, and teardown. Endpoint-oriented mechanisms such as run_evalchemy, run_lm_eval, and run_harbor own only their native task and result contracts. This keeps the lifecycle common without hiding mechanism-specific configuration or outcomes behind a universal do_eval interface.

One EvalGroup (a task set) becomes one EvalchemyResult artifact addressed by evaluation/evalchemy/{model}/{group_id}, so a pipeline picks up exactly the evals it needs and each is cached and reused. The in-loop EvalSuite and the post-hoc EvalGroups draw from the same task menu.

Task sets

Task sets are configured in task_configs.py.

  • CORE_TASKS is the default for in-loop and post-hoc multiple-choice evals.
  • CORE_TASKS_PLUS_MMLU extends CORE_TASKS with MMLU.
  • Named menus (core_evals, key_evals, base_model_evals) bundle task sets into EvalGroups; you can also define custom task lists in task_configs.py and pass them to your own EvalGroups.

In-loop metrics

Beyond task accuracy, the in-loop Levanter evaluator tracks these multiple-choice metrics:

  1. Bits per byte (bpb): bpb = -log_prob / byte_length * ln(2)
  2. Log probability (logprob): raw log probability of the correct answer.
  3. Choice log probability (choice_logprob): log_prob_correct - log(sum(exp(log_prob_i)))
  4. Length-normalized choice probability (choice_prob_norm): exp(log_prob_correct / (byte_length_correct * ln(2))) / sum(exp(log_prob_i / (byte_length_i * ln(2))))

Generation tasks

Generation tasks (for example HumanEval, GSM8K, and MATH) run through the same post-hoc evalchemy path as multiple-choice tasks: the served endpoint answers completion requests, and evalchemy scores them.

Harbor-based evaluation

Agentic launcher runs pass the same RunningModel boundary to run_harbor. Off-cluster sandboxes receive a scoped capability route resolved by the inference session.

  • Harbor supports agent-style benchmarks such as AIME, Terminal-Bench, SWE-bench Verified, and other registry datasets.
  • Marin's Harbor integration supports local Docker and hosted environments such as Daytona, E2B, and Modal.
  • Setup, examples, and environment requirements are documented in Harbor Framework Integration.

Where to go next