Reproduce the additional Delphi seeds¶
experiments/references/train_delphi_seeds.py
records the recipes for additional random seeds at seven Delphi compute budgets
and three seeds each at two additional budgets. It preserves the original
training settings, dataset component names and order, artifact identities, and
Weights & Biases IDs.
The runner defines 22 runs. The Hugging Face release contains 20 checkpoints;
the seed-0 runs at 3e18 and 9e18 are controls and are excluded. Existing
Delphi releases at 1e21, 1e22, and 1e23 are unchanged.
Checkpoint names¶
All model repositories are under marin-community.
For seeds 42 and 62746, append -seed42 or -seed62746 to the base name.
Seed 0 has no suffix. These are training seeds.
| Compute label | Published seeds | HF repository base name |
|---|---|---|
3e18 |
42, 62746 | delphi-3e18-447Mparams-1.2Btokens |
9e18 |
42, 62746 | delphi-9e18-550Mparams-2.9Btokens |
2e19 |
42, 62746 | delphi-2e19-837Mparams-3.6Btokens |
3e19 |
42, 62746 | delphi-3e19-998Mparams-5Btokens |
5e19 |
0, 42, 62746 | delphi-5e19-1.2Bparams-7.4Btokens |
9e19 |
42, 62746 | delphi-9e19-1.4Bparams-10.6Btokens |
2e20 |
42, 62746 | delphi-2e20-1.9Bparams-14.8Btokens |
3e20 |
42, 62746 | delphi-3e20-2.5Bparams-18.6Btokens |
6e20 |
0, 42, 62746 | delphi-6e20-2.9Bparams-29.9Btokens |
The 5e19 and 6e20 labels refer to configured budgets of 5.3e19 and 5.5e20
FLOPs, respectively. Parameter and token counts in repository names are rounded;
the runner records the architecture, batch size, step count, and optimizer for
each budget.
Inputs and setup¶
These commands require access to Marin's Iris cluster, the existing tokenized
GCS caches listed in the runner's DATASETS, and its
meta-llama/Meta-Llama-3.1-8B tokenizer. The runner adopts the existing caches;
it does not rebuild or redistribute them. Set up the checkout and credentials
using the internal onboarding guide.
Run the commands below from the repository root with WANDB_API_KEY exported.
The recorded recipes were recovered from W&B configurations and execution logs
associated with Marin commit 9e946e47cfa74efe43cca1b74360c27f64dd1226.
That identifies the source recipes, not necessarily the revision used to train
each new checkpoint. Use the published model card and original run records for
checkpoint-specific provenance.
Launch the runs¶
Define the submission helper once per shell:
submit_delphi() {
local delphi_label="$1"
shift
uv run iris --cluster=marin job run --no-wait --no-cancel-on-exit \
--priority batch --region us-central2 --cpu 1 --memory 2G --extra cpu \
--job-name "delphi-seeds-${delphi_label}-$(date -u +%Y%m%d-%H%M%S)-$$" \
-- python -m experiments.references.train_delphi_seeds "$@"
}
Submit the 16 original-recipe runs, including the two seed-0 controls:
(
set -e
submit_delphi single-host --slugs 3e18 9e18 2e19 3e19
for delphi_slug in 9e19 2e20 3e20; do
for delphi_seed in 42 62746; do
submit_delphi "${delphi_slug}-seed${delphi_seed}" \
--slugs "$delphi_slug" --seeds "$delphi_seed"
done
done
)
Submit the six runs at the additional compute budgets separately:
(
set -e
submit_delphi 5e19 --slugs 5e19
for delphi_seed in 0 42 62746; do
submit_delphi "6e20-seed${delphi_seed}" \
--slugs 6e20 --seeds "$delphi_seed"
done
)
Each multi-host run has its own top-level Iris job to avoid JAX coordinator
name collisions. The single-host runs share a top-level job within each group.
Use --slugs and --seeds to select runs; --max-concurrent controls executor
concurrency within one top-level job and defaults to 16.
Before retrying a run, stop its previous coordinator. Successful artifacts are
reused, and interrupted runs resume from their own checkpoints. The unique
submission job names do not change training identities. VERSION = "2026.09.22"
fixes the recorded artifact identities; use a new version for a new recipe or
an independent rerun that must not reuse those artifacts.
Outputs and HF export¶
The artifact name is delphi-seed-repeats-<slug>-seed<seed>, and the trainer
and W&B run ID is that name followed by -2026.09.22. Under the configured
Marin storage prefix, the final HF export is:
The model is Qwen3. Its historical HF reference checkpoint can leave
architectures: ["LlamaForCausalLM"] in the exported config.json.
Before publishing a fresh export, change only that field to
"architectures": ["Qwen3ForCausalLM"]. This is a metadata correction;
it does not convert or change the weights. The released checkpoints already
contain the corrected field.
Publish the config, tokenizer files, safetensors files, and shard index when
present. Executor files such as .artifact.json, .executor_info, and
.executor_status are not inference inputs. The models use Apache-2.0, matching
the existing Delphi release.