marin-community/marin

Marin – Open‑source platform for building foundation models

What it is

• A research‑grade software stack (and community) focused on the whole lifecycle of large language models: data curation, tokenisation, pre‑training, post‑training, and evaluation.
• Designed to be open development: every experiment, configuration and even failed runs are recorded and published.

Core capabilities

Area	What Marin provides
Experiment definition	Declarative, step‑based pipelines (similar to a Makefile) that describe data preparation, training, and evaluation. Steps are lazily materialised and can depend on one another.
Scaling suite – Delphi	A recipe that maps a compute budget to model size, a set of scaling‑law experiments run on Google TPU Research Cloud, and code to reproduce the mixture‑of‑experts (MoE) pipelines. Checkpoints and plot‑ready data are hosted on Hugging Face.
Mixture‑of‑Experts support	Tools for deterministic mixing of public datasets (Nemotron‑CC, StarCoderData, ProofPile 2) and for MoE load‑balancing (quantile‑balancing).
Optimisation & training utilities	Configurable Adam variants (CompletedAdamHParams), resource specifications (ResourceConfig), and ready‑made model definitions (e.g., llama_nano).
Extensible agent skills	Loadable “skills” (e.g., add_scaling_heuristic, add-dataset) that automate common workflow steps.
Documentation & tutorials	Full docs on ReadTheDocs, quick‑start guides for training a tiny model, and larger‑scale examples (1 B‑parameter DCLM, 8 B and 32 B models).

Typical workflow (tiny‑model example)

# 1️⃣ Tokenise a dataset lazily
tinystories = tokenized(
    name="tokenized/tinystories",
    source="roneneldan/TinyStories",
    tokenizer=marin_tokenizer,
    sample_count=1_000,
)

# 2️⃣ Define a training step that depends on the tokenised data
nano_model = train_lm(
    name="checkpoints/marin-nano-tinystories",
    version="v1",
    model=llama_nano,
    optimizer=AdamConfig(learning_rate=6e-4, weight_decay=0.1),
    datasets={tinystories: 1.0},
    batch_size=4,
    seq_len=2048,
    num_train_steps=100,
    resources=ResourceConfig.with_cpu(),
)

# 3️⃣ Execute the pipeline
StepRunner().run([lower(nano_model)])

The same pattern scales to multi‑GPU/TPU pods and to mixtures of many datasets.

Getting started

1. Read the installation guide: docs/tutorials/installation.md.
2. Run the tiny‑model tutorial (docs/tutorials/first-experiment.md).
3. Follow the larger‑scale training guide (docs/tutorials/train-an-lm.md).
4. Browse the experiment catalogue (docs/reports/index.md).
5. Join the community on Discord for support and collaboration.

Community & backing

• Core contributors are from Stanford CRFM and Open Athena.
• Supported by Google TPU Research Cloud, the Jen‑Hsun & Lori Huang Foundation, Siegel Family Endowment, and Schmidt Sciences.

Why it matters Marin aims to make frontier‑scale LLM research reproducible and transparent. By open‑sourcing the full training stack—including data pipelines, scaling laws, and MoE balancing—researchers can build, compare, and extend large models without reinventing the underlying infrastructure.

---

Links

• 📚 Documentation: https://marin.readthedocs.io
• 🐙 Source code: https://github.com/marin-community/marin
• 🤖 Model checkpoints (Delphi): https://huggingface.co/collections/marin-community/delphi-69f93cbd09845c03b070bae9
• 💬 Discord: https://discord.gg/J9CTk7pqcM