Skip to content

Running Evaluations with Marin

The shared evaluation launcher runs Evalchemy and Harbor evaluations against a registered model or a file-backed model configuration. It starts one OpenAI-compatible model server, runs every selected evaluation against that endpoint, writes one durable record per evaluation, and tears the server down.

Use the launcher for routine post-hoc evaluations. Use the composable APIs later in this page when an evaluation must be part of a Marin pipeline or consume an ArtifactStep[LevanterCheckpoint]. See Evaluation Overview for the evaluator and serving architecture.

Prerequisites

  • Run commands from the Marin repository with uv.
  • Use a model key registered in experiments/evaluation/models.py or experiments/evaluation/serve/models/, or supply a file with the model catalog schema.
  • Have access to the selected Iris cluster and its TPU or GPU serving resources.
  • For Harbor evaluations, use an environment with access to the Daytona credential described in Harbor credentials.

Launch registered models and evaluations

The command accepts one model key and either a named suite or comma-separated evaluation keys:

uv run python -m experiments.evaluation.cli launch \
  --model <model-key> \
  --evals <suite-or-eval-keys>

Run the resolved plan before submitting an unfamiliar model or suite. --dry-run prints the model location, serving backend, accelerator, target cluster or region, priority band, task names, and records prefix:

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-0.6b \
  --evals smoke \
  --dry-run

Launch file-backed evaluations

Use --model-config for a one-off model export that does not belong in the central catalog. The file uses the same YAML or JSON schema as experiments/evaluation/serve/models/; its name becomes the served model name and run identity. This dry run evaluates a fresh RL export without modifying the checkout:

uv run python -m experiments.evaluation.cli launch \
  --model-config /path/to/fresh-rl-checkpoint.yaml \
  --evals smoke \
  --dry-run

Exactly one of --model <registry-key> or --model-config <path> is required. record.json stores the normalized model schema under model.config, including registry-backed launches, so the durable record is self-contained.

Use --evalchemy-config for an Evalchemy or lm-eval definition that does not belong in the central registry. This dry run resolves the checked-in IFEval definition without opening Iris:

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-8b \
  --evalchemy-config experiments/evaluation/configs/evalchemy/ifeval.yaml \
  --dry-run

The option is repeatable. It composes with registry selections and Harbor policy files, so one served model can run all three sources:

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-8b \
  --evals mmlu-smoke \
  --evalchemy-config experiments/evaluation/configs/evalchemy/ifeval.yaml \
  --harbor-config experiments/evaluation/configs/harbor/aime-smoke.yaml \
  --limit 2 \
  --dry-run

Each config file is one evaluation with its own record.json. Marin decodes the launch YAML without importing Evalchemy, then the evaluation child invokes the evalchemy console script from the pinned external runtime. A dry run checks the YAML shape and launch plan; unavailable task names fail when the Evalchemy process starts. The record keeps the normalized launch-time task routing, few-shot counts, generation arguments, concurrency, context limit, and instance limit under eval.tasks and eval.evalchemy. Its provenance stores the exact Evalchemy requirement and optional runtime extras.

Command recipes

Run a capped Evalchemy smoke on the smallest registered Qwen model:

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-0.6b \
  --evals smoke

Run the 14-task NLP suite on a base model. This suite works with base and instruct models:

uv run python -m experiments.evaluation.cli launch \
  --model llama-3.1-8b-base \
  --evals nlp \
  --version 2026.07.27 \
  --description "Llama 3.1 8B base NLP baseline"

Run the chat-template benchmarks on an instruct model:

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-8b \
  --evals chat

Run only MMLU and GSM8K, cap each evaluation at 32 instances, and return after submission:

uv run python -m experiments.evaluation.cli launch \
  --model llama3.1-8b-instruct \
  --evals mmlu,gsm8k \
  --limit 32 \
  --no-wait

Run a two-task Terminal-Bench 2 Harbor check on the validated Qwen3-32B GPU shape:

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-32b \
  --evals tb2-lite

Route a GPU serve group to RNO2A at interactive priority:

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-32b \
  --evals tb2-lite \
  --federated_cluster cw-rno2a \
  --priority interactive

Run one capped Evalchemy task and one capped Harbor benchmark against the same model server:

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-32b \
  --evals mmlu,tb2-lite \
  --limit 2

Run the full Terminal-Bench 2 preset or all standard Harbor presets:

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-32b \
  --evals tb2 \
  --no-wait

uv run python -m experiments.evaluation.cli launch \
  --model qwen3-32b \
  --evals agentic \
  --no-wait

agentic runs seven full Harbor datasets and can consume substantial Daytona and inference capacity. Use tb2-lite, swebench-lite, aime-smoke, or --limit N for validation runs.

Run one OT-TBLite trial with the registered Grug model and OpenCode agent policy:

uv run python -m experiments.evaluation.cli launch \
  --model grug-agentic-s3-step1903 \
  --evals ot-tblite \
  --limit 1

The qwen3-32b / tb2-lite path follows the H100x2 acceptance run recorded in issue #6503. Harbor trial restore and persistence use the selected GCS or S3 records store after PR #7625.

Object-store model loading

CUDA vLLM serves s3:// checkpoints with RunAI model streamer 0.16.1. The launcher gives the streamer's low-speed check 10 seconds and sends its warning and error logs to the Iris job log. GPU object-store loads default to distributed RunAI reads: ranks divide the reads and share the data. Explicit --load-format or --model-loader-extra-config arguments override this default.

The following variables may prefix an evaluation command and are forwarded to the remote inference child:

  • RUNAI_STREAMER_S3_REQUEST_TIMEOUT_MS overrides the 10,000 ms low-speed window.
  • RUNAI_STREAMER_CONCURRENCY controls the number of object-store reader workers. Try 4 if an S3 endpoint fails under the default concurrency of 8; this may increase model-load time.
  • RUNAI_STREAMER_S3_MAX_INFLIGHT_MIB caps in-flight data per S3 client on RunAI 0.16.1. Leave the derived default in place unless concurrency reduction is insufficient.
  • RUNAI_STREAMER_CHUNK_BYTESIZE changes the default 8 MiB object-store chunk. Benchmark changes; larger chunks reduce request count and increase memory per request.
  • RUNAI_STREAMER_LOG_TO_STDERR, RUNAI_STREAMER_LOG_LEVEL, and RUNAI_STREAMER_S3_TRACE control diagnostics. S3 trace logging writes a file and is intended for a bounded reproduction.

For example, retry a model-load failure with half the default reader concurrency:

RUNAI_STREAMER_CONCURRENCY=4 \
uv run python -m experiments.evaluation.cli launch \
  --model grug-agentic-s3-step1903 \
  --evals ot-tblite \
  --limit 1

Do not set AWS_RETRY_MODE or AWS_MAX_ATTEMPTS to increase retries for this loader. The AWS CRT client bundled with RunAI already retries each request five times, and that build does not propagate AWS_MAX_ATTEMPTS to the CRT retry count. Marin separately retries vLLM startup up to three times when RunAI reports a transient read failure.

Named suites

Suite Evaluator Contents and constraints
smoke Evalchemy Capped MMLU abstract algebra and GSM8K; use before a larger run.
core Evalchemy Eleven OpenLLM-style, generation, code, and MATH500 evaluations. Use a model with a chat template.
nlp Evalchemy Fourteen log-likelihood and greedy NLP evaluations; supports base and instruct models.
chat Evalchemy MATH500, AIME24, and OlympiadBench; requires a chat-template model.
math Evalchemy MATH500, AIME24, and zero-shot GSM8K; requires a chat-template model.
agentic Harbor Terminal-Bench 2, SWE-bench, GAIA, BFCL, Aider, MedAgentBench, and FinanceAgent in Daytona.

The code suite is registered but is not runnable with the current pinned evaluation image because its HumanEvalPlus and MBPPPlus dependencies are absent. Individual evaluation keys are defined in experiments/evaluation/evals.py. Useful Harbor keys outside agentic include tb2-lite, swebench-lite, swebench-full, aime-smoke, aime-harbor, and ot-tblite.

Choose a model

--model takes a registry key, not an arbitrary Hugging Face repository:

Model key Model type and default placement Typical selection
qwen3-0.6b Instruct, TPU smoke
llama-3.1-8b-base Base, TPU nlp
llama3.1-8b-instruct Instruct, TPU core, nlp, or chat
qwen3-32b Instruct, CoreWeave H100x2 tb2-lite, tb2, or agentic
grug-agentic-s3-step1903 Instruct export, CoreWeave H100x8 ot-tblite

The complete catalog is the union of experiments/evaluation/models.py and YAML files under experiments/evaluation/serve/models/. The model catalog schema documents how to register a Hugging Face model or object-store checkpoint.

Launch controls

  • --evals smoke selects a named suite. --evals mmlu,gsm8k selects explicit keys. One invocation may mix Evalchemy and Harbor keys.
  • --evalchemy-config path.yaml adds one portable Evalchemy or lm-eval definition. Repeat the option for multiple files.
  • --harbor-config path.yaml adds one Harbor policy. Repeat the option for multiple files.
  • --limit N overrides the configured instance cap for every selected evaluation.
  • --no-wait returns after Iris submission. Without it, the command waits for terminal records and prints their metrics.
  • --platform tpu|gpu overrides the model's default platform when the model resource hint supports that platform.
  • --accelerator v6e-8 or --accelerator H100x8 requests an exact compatible slice.
  • --federated_cluster cw-rno2a overrides the GPU fleet profile's target cluster. The launcher always submits through the marin Iris controller.
  • --priority production|interactive|batch sets the orchestrator and serve priority band. Omitting it preserves Iris priority inheritance.
  • --version and --description attach run metadata.
  • --records-prefix changes the result store.

Each invocation serves the model once and evaluates the selected keys in order. An evaluation failure gets its own terminal record and does not skip later evaluations. An inference failure records the current and remaining evaluations as infrastructure failures.

Harbor credentials

Daytona-backed definitions first use DAYTONA_API_KEY from the launch environment, then use the approved DAYTONA_EVAL_API_KEY version in Google Secret Manager. Do not put resolved credentials in model YAML, evaluation definitions, or command arguments. Evalchemy-only runs do not require a Daytona credential. See Harbor evaluation for dataset, agent, and endpoint details.

Results

TPU-routed runs default to gs://marin-eval-metadata/evals. CoreWeave GPU runs default to s3://marin-us-east-02a/marin/evals. --dry-run prints the effective prefix.

Every selected evaluation writes {records_prefix}/{run_id}/record.json and a FineStore archive under its results directory. Evalchemy writes its aggregate JSON and per-task JSONL as FineStore source artifacts and writes normalized samples for every extraction filter. Marin derives one evaluator-neutral rollouts_v1 conversation per (task, doc_id, trial_id) attempt from those samples, deduplicated across extraction filters. Harbor preserves its job metadata, trial results, and trajectories while writing normalized agentic samples and steps; Marin derives its rollouts_v1 rows from steps. The evaluator's raw artifacts remain in FineStore. Harbor's job tree remains the checkpoint used for resume and inspection. Evalchemy records include the normalized launch configuration; Harbor records include the dataset, agent, environment, task limit, and source-policy digest. A Harbor infrastructure exception remains ungraded and counts against the completion gate after its artifacts are saved; a verifier-scored zero remains a completed evaluation with a zero score.

Evaldash indexes records from both default stores. The record is the source of truth for model, evaluation identity, status, metrics, hardware, provenance, and Iris job paths.

Use the shared launcher in a pipeline

experiments.evaluation.pipeline.eval_step wraps the same registered model and evaluation selection in a lazy artifact. Save the pipeline as eval_pipeline.py:

from experiments.evaluation.models import models
from experiments.evaluation.pipeline import eval_step
from marin.execution.step_runner import StepRunner

step = eval_step(
    models()["qwen3-1.7b"],
    "smoke",
    version="2026.07.27",
    limit=32,
)
StepRunner().run([step.lower()])

Run the pipeline itself inside an Iris job so eval_step can submit its orchestrator job:

uv run iris --cluster marin job run -- python eval_pipeline.py

The step uses the same serving, executor, record, and result paths as the CLI. Its artifact identity includes the model, evaluation selection, limit, and version.

Compose Evalchemy steps from Levanter checkpoints

The lower-level post-hoc API accepts an ArtifactStep[LevanterCheckpoint]. Use the return value of a training run, or adopt an existing checkpoint with ArtifactStep.adopt. These APIs compose Evalchemy task groups directly; Harbor evaluations use the shared launcher above.

Core APIs

Reusable step builders live in marin.experiment.evaluation; experiment task menus remain under experiments/evals:

from marin.evaluation.hardware import AcceleratorChoice
from marin.evaluation.model_config import ServeConfig
from marin.experiment.evaluation import (
    EvalGroup,
    eval_step,
    eval_steps,
    eval_report,
)
from experiments.evals.evals import (
    core_evals,
    key_evals,
    base_model_evals,
)
  • An EvalGroup is one set of tasks evaluated against one served model. It is the composable unit: each group becomes one artifact, addressed by evaluation/evalchemy/{model}/{group_id}.
  • eval_step(model, group) builds one eval artifact; eval_steps(model, groups) builds one per group.
  • eval_report(results, name=...) aggregates a suite's results into one EvalReport artifact.
  • core_evals / key_evals / base_model_evals are named menus — lists of EvalGroup drawn from experiments/evals/task_configs.py, the same task menu the in-loop EvalSuite on train_lm uses.

Each helper returns an ArtifactStep. Build them with run(...) (which returns the loaded artifacts) or with StepRunner().run([lower(x) for x in steps]).

Run a named suite

from marin.execution.lazy import ArtifactStep, run
from marin.training.training import LevanterCheckpoint

from marin.evaluation.hardware import AcceleratorChoice, Platform
from marin.experiment.evaluation import eval_report, eval_steps
from experiments.evals.evals import key_evals

# Adopt a pre-existing checkpoint as a typed handle (no copy, no recompute). A relative source
# resolves against the local bucket (MARIN_PREFIX, set by iris); pass an absolute gs:// path to pin.
model = ArtifactStep.adopt(
    "perplexity-models/llama-200m",
    "2026.06.30",
    "gcsfuse_mount/perplexity-models/llama-200m",
    kind=LevanterCheckpoint,
)

accelerator = AcceleratorChoice(platform=Platform.TPU, tpu_type="v6e-8")
results = eval_steps(model, key_evals(accelerator=accelerator))
report = eval_report(results, name=f"{model.name}/key")

if __name__ == "__main__":
    run(report)

key_evals returns two groups: a generation group over KEY_GENERATION_TASKS and a multiple-choice group over KEY_MULTIPLE_CHOICE_TASKS. eval_report depends on both and materializes the merged per-task metrics.

core_evals and base_model_evals follow the same shape. base_model_evals runs CORE plus each MMLU cut as its own group, so every cut is evaluated (each group has a distinct identity).

Compose your own groups

An EvalGroup states its tasks, its serving backend, and its id explicitly:

from marin.execution.lazy import run

from marin.evaluation.evalchemy.runner import EvalchemyRunConfig
from marin.evaluation.hardware import AcceleratorChoice, Platform
from marin.evaluation.model_config import ServeConfig
from marin.experiment.evaluation import EvalGroup, eval_report, eval_steps
from experiments.evals.task_configs import CORE_TASKS, KEY_GENERATION_TASKS

accelerator = AcceleratorChoice(platform=Platform.TPU, tpu_type="v6e-8")
groups = [
    EvalGroup(
        config=EvalchemyRunConfig(name="core", tasks=CORE_TASKS),
        accelerator=accelerator,
    ),
    EvalGroup(
        config=EvalchemyRunConfig(name="generation", tasks=KEY_GENERATION_TASKS, max_gen_toks=4096),
        serve=ServeConfig(),
        accelerator=accelerator,
    ),
]

report = eval_report(eval_steps(model, groups), name=f"{model.name}/custom")

if __name__ == "__main__":
    run(report)

EvalchemyRunConfig.name is the task-group segment of the artifact name. Choose a stable name per group.

Read results

resolve (or run) returns the typed artifacts. An EvalReport carries the merged metrics:

from marin.execution.lazy import resolve

report_artifact = resolve(report)
print(report_artifact.task_metrics)  # {task: {metric: value}}
print(report_artifact.averages)      # suite-level rollups

Each individual result is a FineStoreEvalchemyResult; task_metrics() reads the per-task scores from Evalchemy's aggregate source artifacts in FineStore.

Run the repository example scripts

The checked-in examples use the lower-level API for adopted Levanter checkpoints. They are deferred-version CLIs (see marin.experiment.cli): --version supplies the run-wide version, the plan prints without --run, and --run builds it.

# print the plan (no build)
uv run python -m experiments.evals.run_key_evals --version dev
# build it; --limit caps examples per task for a fast cluster smoke
uv run python -m experiments.evals.run_key_evals --version dev --run --limit 5
uv run python -m experiments.evals.run_base_model_evals --version 2026.07.16 --run

They adopt a checkpoint, build the suite, compile a report, and log results to W&B. --version dev resolves the eval artifacts to a mutable version that rebuilds every run — pass a calendar version to pin a run.

Serve a composed group on GPU

AcceleratorChoice selects TPU or GPU placement. ServeConfig contains server behavior and is independent of placement. The GPU path runs the Marin CUDA vLLM fork in an isolated uvx environment, which runs without nvcc. Set vllm_extra_args for models that would otherwise JIT-compile a kernel at warmup.

Qwen gated-delta-net models (qwen_gdn_linear_attn: Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3-Next-80B-A3B) are the current case. Their default FlashInfer GDN prefill kernel is JIT-compiled, so without a compiler the serve child dies at warmup (Could not find nvcc) and never registers its endpoint. Pass --gdn-prefill-backend triton to use the compiler-free triton backend:

EvalGroup(
    config=EvalchemyRunConfig(name="core", tasks=CORE_TASKS),
    serve=ServeConfig(
        tensor_parallel_size=8,
        vllm_extra_args=("--gdn-prefill-backend", "triton"),
    ),
    accelerator=AcceleratorChoice(
        platform=Platform.GPU,
        gpu_type="H100",
        gpu_count=8,
    ),
)

Other GPU-served models (DeepSeek-V2-Lite, Qwen3-30B-A3B, …) need no extra flags. See marin.evaluation.model_config.ServeConfig and marin.evaluation.hardware.AcceleratorChoice for the two contracts.

Parameter reference

EvalGroup

  • config: the EvalchemyRunConfig, including tasks, generation settings, concurrency, and limits.
  • serve: the model-server behavior (vllm or levanter and its server options).
  • resource_hint: optional CPU, memory, and disk overrides for the inference worker. Host memory defaults to a size derived from the checkpoint's weight files.
  • accelerator: the exact TPU or GPU slice used for inference.
  • tokenizer: HF tokenizer id the eval client loads; defaults to the served checkpoint. Set it to a base-model HF id when serving a gs:// path the Evalchemy client cannot load a tokenizer from.
  • discover_latest_checkpoint: whether to resolve the latest HF checkpoint under the model path.

eval_report

  • results: the EvalResult steps to aggregate.
  • name: the report's identity segment (evaluation/report/{name}).

For deeper dives, see:

  • Evaluation Overview
  • experiments/evaluation/README.md
  • experiments/evaluation/evals.py
  • experiments/evaluation/models.py
  • experiments/evals/task_configs.py
  • experiments/evals/evals.py