Marin

"I am not afraid of storms, for I am learning how to sail my ship."
– Louisa May Alcott

Marin is a research program, software platform, and community for the research and development of foundation models.

Marin's concern is training large language models. This includes data curation, transformation, filtering, tokenization, pretraining, posttraining, and evaluation. Beyond the artifacts, software, and infrastructure, behind these models, Marin is committed to openly sharing all of the process knowledge required to build these models.

Marin's core value is open development. We document our processes, experiments, and decisions as they happen. Every step, from raw data to the final model, is recorded. Failed experiments are part of that record.

Marin has also been used for building audio-text models, DNA, and protein models. We encourage this work through the use of Marin as a library, in marin/experiments.

Current work

Frontier mixture-of-experts

Our current focus is pretraining, from scratch, and posttraining a large (5e24 model-FLOPs, 500 billion+ total parameters) mixture-of-experts model to succeed on tasks of importance to scientists and researchers.

Scaling suite

Delphi is Marin's open scaling suite scaling a LLM recipe from 3e18 to 1e23 FLOPs, inspired by Pythia. It has three parts: a scaling recipe that maps compute budgets to model configurations, a scaling suite trained from that recipe on the Google TPU Research Cloud, and a scaling law that uses the smaller Delphi models to predict the larger ones.

We released:

• Checkpoints for every run, available on Hugging Face at marin-community/delphi
• Training mixture pipelines that deterministically reproduce the mix from the public Nemotron-CC, StarCoderData, and ProofPile 2 in the Marin repo
• Recipe code as a forkable CompletedAdamHParams class in the Marin repo
• Development methodology as the add_scaling_heuristic agent skill in the Marin repo
• Plot-ready data for the Delphi figures, with one config per figure and a wandb_url on every row, at marin-community/delphi-blog-data

Progress was tracked in GitHub issue #1337.

Other learnings

Some additional consolidated learnings can be found on the Open Athena blog. A selection, below:

• Cluster Scheduling with Iris · scheduling jobs across heterogeneous clusters
• Improving our LLM Pretraining Efficiency · squeezing more throughput from pretraining
• Scaling Laws That Extrapolate 300× Past the Fit (Delphi) · predicting big models from small
• Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scale · keeping MoE experts load balanced

Other models

Previously, we used Marin to train an 8B parameter model that outperformed Llama 3.1 8B on our base-model benchmark suite. You can see the training script or read the retrospective. We also trained Marin 32B.

Quick Links

• Installation
• First Experiment
• Local GPU Setup
• The Language Modeling Pipeline
• Experiment Reports

Documentation Structure

Our documentation is organized into the following main sections:

• Tutorials: Step-by-step guides to help you get started with Marin, including installation, basic usage, and local GPU setup
• Explanation: Background information and context about the project
• Experiment Reports: Reports from our experiments
• Developer Guide: Information for developers who want to contribute to Marin
• Technical Reference: Detailed technical information about Marin's architecture and components

These sections are available on the left side bar (or hamburger menu).

Get Involved

To get started with Marin:

• Install Marin.
• Train a tiny language model using Marin.
• See how to run a much larger DCLM 1B/1x experiment using Marin.
• See a summary of the experiments we've run.
• Join the Marin Discord to chat with the community.

Get Help

If you have any questions or need help, please feel free to reach out to us on Discord or open an issue on GitHub.