OpenMed CodonRoBERTa multi-species mRNA language models release

TL;DR – OpenMed released an end‑to‑end protein‑engineering pipeline that adds a codon‑optimization stage powered by a new family of transformer language models, with CodonRoBERTa‑large‑v2 achieving a perplexity of 4.10 and a CAI Spearman correlation of 0.404, and a multi‑species suite covering 25 organisms trained in only 55 GPU‑hours (≈ $165). The suite includes a universal 311 M‑parameter model and three specialist models (human, E. coli, CHO) that outperform prior approaches and enable rapid generation of expression‑ready DNA.

1. What the Release Contains

Component Description Key Result
Protein folding ESMFold v1 predictions on 30 protein chains Average PTM = 0.79 (high topology confidence)
Sequence design ProteinMPNN on scaffold 7K00 42 % amino‑acid recovery
mRNA optimization New transformer models trained on 250 k E. coli CDS, then scaled to 381 k CDS across 25 species CodonRoBERTa‑large‑v2: perplexity 4.10, CAI Spearman 0.404; 4 production models covering 25 organisms trained in 55 GPU‑hours

The folding and design stages reuse established open‑source tools (ESMFold, ProteinMPNN). The codon‑optimization stage is entirely new: a suite of RoBERTa‑style language models trained on codon‑level sequences with species‑conditioning.

2. Architecture Exploration – Which Transformer Works Best for Codons?

Model Parameters Architecture Training data (250 k E. coli CDS)
CodonBERT (baseline) 6 M BERT‑tiny (6 layers) –
ModernBERT‑base 90 M ModernBERT (22 layers, RoPE) –
CodonRoBERTa‑base 92 M RoBERTa (12 layers) –
CodonRoBERTa‑large 312 M RoBERTa (24 layers) –
CodonRoBERTa‑large‑v2 312 M RoBERTa (24 layers, refined LR & warm‑up) –

Results on the E. coli test set

Model Perplexity CAI Spearman Synonymous recovery
CodonRoBERTa‑large‑v2 4.10 0.404 7.7 %
CodonRoBERTa‑base 4.01 0.219 8.5 %
CodonRoBERTa‑large 4.01 0.025 7.6 %
ModernBERT‑base 26.24 0.070 8.5 %
CodonBERT (baseline) 17.18 –0.629 0 %

Takeaways

  1. RoBERTa outperforms ModernBERT by a factor of six on perplexity and by a factor of six on CAI correlation, indicating that the classic RoBERTa MLM architecture is better suited to the statistical structure of codon sequences.
  2. Pre‑training on English text harms performance – initializing ModernBERT from an English checkpoint degraded its ability to learn codon patterns.
  3. Hyper‑parameter tuning matters more than size – the v2 checkpoint (lr 5e‑5, warm‑up 2 k steps) achieved a 16‑fold improvement in CAI correlation despite a marginally higher perplexity.
  4. The 92 M‑parameter base model is a practical low‑cost alternative – it matches perplexity of the large model while using one‑third the parameters.

3. End‑to‑End Pipeline

3.1 Protein Folding with ESMFold

  • Tool: Meta’s ESMFold v1 (single‑sequence, no MSA).
  • Performance on 30 PDB chains: average PTM = 0.79, per‑residue pLDDT ≈ 34 (lower on multi‑chain targets). Folding takes 10–30 s per protein on an A100.

3.2 Sequence Design with ProteinMPNN

  • Tool: ProteinMPNN (graph‑based inverse folding).
  • Result on scaffold 7K00: generated three sequences; the best recovered 42 % of the native amino acids at temperature 0.1.

3.3 mRNA Optimization with CodonRoBERTa

  • Problem – Traditional CAI tables replace each codon independently, ignoring context and leading to repetitive, sometimes sub‑optimal sequences.
  • Solution – Masked Language Modeling (MLM) on codon tokens learns contextual codon preferences. The model predicts masked codons from surrounding context, capturing long‑range dependencies.
  • Evaluation metrics
    • Perplexity – 4.10 for CodonRoBERTa‑large‑v2 (≈ 4 equally likely codons per mask).
    • CAI Spearman correlation – 0.404 (p < 10⁻²⁰), indicating strong alignment with biologically preferred codon usage.
    • Synonymous recovery – 12.1 % top‑1 synonymous prediction, showing the model respects amino‑acid constraints.
  • Usage example (Python, Hugging Face Transformers):
    from transformers import RobertaForMaskedLM
model = RobertaForMaskedLM.from_pretrained("OpenMed/CodonRoBERTa-large-v2")
tokenizer = CodonTokenizer()  # 69‑token codon vocab
seq = "ATG GCT AAA GGT ..."
inputs = tokenizer(seq, return_tensors="pt")
with torch.no_grad():
    scores = model(**inputs).logits

4. Scaling to 25 Species – A Single Model that Knows Every Organism

4.1 Data Engineering

  • Downloaded RefSeq CDS for 25 organisms (19 bacteria, 3 yeasts, 3 mammals).
  • Validation steps: start/stop codons, length % 3 = 0, no internal stops.
  • Final dataset: 381 283 sequences (≈ 3 GB), split 95 %/5 % train/test per species.

4.2 Species‑Conditioned Tokenizer

  • Extended the 69‑token codon vocabulary with 25 special species tokens, yielding a 94‑token vocabulary.
  • Each sequence is prefixed with its species token (e.g., [HUMAN]), enabling the model to learn organism‑specific codon biases while sharing parameters across all species.

4.3 Universal Base Model

  • Architecture: RoBERTa‑large (311.9 M params) with the 94‑token vocab.
  • Training: 48 h on 4 × A100 GPUs, 50 k steps (≈ 4.5 epochs), bf16 mixed precision, Fully Sharded Data Parallel (FSDP).
  • Test perplexity: 24.9 – higher than the single‑species model because the model must capture 25 distinct codon usage patterns.

4.4 Species‑Specific Fine‑Tuning

Species Train sequences Fine‑tune steps GPU‑hours Test perplexity
Human (therapeutic mRNA) 131 k 15 k 4 h 24.3
E. coli (bacterial expression) 8.5 k 5 k 0.5 h 25.3
CHO (biopharma mammalian) 42.5 k 10 k 2.5 h 25.5

The human specialist improves over the universal base (2.4 % lower perplexity) despite using far fewer parameters, confirming the benefit of transfer learning from a multi‑species foundation.

4.5 Model Suite (released on Hugging Face)

  • OpenMed/CodonRoBERTa-large-multispecies – universal 311 M‑param model.
  • OpenMed/CodonRoBERTa-large-human – specialist for human cells (best overall).
  • OpenMed/CodonRoBERTa-large-ecoli – specialist for E. coli.
  • OpenMed/CodonRoBERTa-large-cho – specialist for CHO cells.
  • OpenMed/CodonRoBERTa-large-v2 – single‑species E. coli model with the lowest perplexity (4.10) and highest CAI correlation (0.404).
  • OpenMed/CodonRoBERTa-base – 92 M‑param efficient alternative.

All models are Apache‑2.0 licensed, stored in safetensors, and can be loaded with from_pretrained().

5. End‑to‑End Workflow Example

  1. Fold – Use ESMFold to obtain a 3‑D structure for the target protein.
  2. Design – Run ProteinMPNN, fixing functional residues, to generate candidate amino‑acid sequences.
  3. Validate – Re‑fold candidates with ESMFold; keep those with high pLDDT/PTM.
  4. Optimize – Pass the chosen amino‑acid sequence to the appropriate CodonRoBERTa specialist (e.g., CodonRoBERTa-large-human) to obtain a codon‑optimized DNA string.
  5. Synthesize – Order the DNA, clone, and test expression in the chosen host.

The entire computational loop runs on a single GPU in under an hour for a typical protein, compressing a process that historically required weeks of wet‑lab iteration.

6. Position in the Landscape & Future Directions

6.1 Comparison to Contemporary Models

Model Params Data size Species conditioning Reported CAI / translation‑efficiency metric
OpenMed CodonRoBERTa 312 M 381 k CDS (25 species) Single model with 25 tokens CAI Spearman 0.404 (human specialist)
mRNABERT (Xiong et al., 2025) 86 M 18 M sequences No species token R² 0.66 on translation efficiency
NUWA (Zhong et al., 2026) – 115 M sequences, ~25 k species Three domain‑specific models Beats CodonBERT on 11/13 benchmarks

OpenMed’s unique contributions

  • A single, species‑conditioned model that can be prompted for any of the 25 covered organisms, unlike NUWA’s three separate domain models.
  • Demonstrated transfer learning: fine‑tuning the universal base on as few as 8.5 k E. coli CDS yields a specialist that outperforms the base.
  • Provides a complete, runnable pipeline (fold → design → codon‑optimize) with all code and weights released under permissive licenses.

6.2 In‑Progress Research – CodonJEPA

  • Goal – Replace token‑level MLM with Joint Embedding Predictive Architecture (JEPA) to make synonymous codons indistinguishable in embedding space.
  • Early results (15 k steps, same data):
    • Synonymous robustness (cosine similarity) = 0.9997 for JEPA vs 0.9414 for MLM.
    • Dimensional collapse observed (91 % variance in a single component); ongoing work on stronger VICReg regularization.
  • If collapse is resolved, JEPA could provide codon embeddings that are inherently amino‑acid aware, a capability MLM cannot achieve.

6.3 Roadmap (next 12 months)

  1. Scale CodonRoBERTa on the public 36 M‑sequence mRNABERT dataset and add contrastive alignment with ProtT5‑XL.
  2. Extend fine‑tuning to additional organisms (yeast, mouse, Pichia) and publish updated specialist checkpoints.
  3. Finalize CodonJEPA by fixing collapse, benchmark against mRNABERT, and integrate as an optional embedding layer in the pipeline.
  4. Enrich the pipeline with RFdiffusion for de‑novo backbone generation, and add expression‑stability prediction heads.

7. Practical Deployment

  • Inference hardware – Single A100 (≥ 16 GB VRAM) or any GPU with ≥ 12 GB for the 92 M‑param base model.
  • Software stack – PyTorch 2.5+, transformers ≥ 4.40, flash‑attn2 for efficient attention, FSDP for training.
  • Licensing – All components (ESMFold, ProteinMPNN, OpenMed models) are MIT or Apache‑2.0, allowing commercial use.
  • Cost – The entire training effort cost ≈ $165 on AWS spot instances (4 × A100, 55 GPU‑hours).

8. References

  • Jumper, J. et al. “Highly accurate protein structure prediction with AlphaFold.” Nature (2021).
  • Lin, Z. et al. “Evolutionary‑scale prediction of atomic‑level protein structure with a language model.” Science (2023).
  • Dauparas, J. et al. “Robust deep learning‑based protein sequence design using ProteinMPNN.” Science (2022).
  • Cheng, J. et al. “CodonBERT: a language model for codon optimization.” Nucleic Acids Research (2024).
  • Xiong, Y. et al. “mRNABERT: advancing mRNA sequence design with a universal language model and comprehensive dataset.” Nature Communications (2025).
  • Zhong, Y. et al. “Large mRNA language foundation modeling with NUWA for unified sequence perception and generation.” bioRxiv (2026).
  • Warner, B. et al. “ModernBERT: Smarter, Better, Faster, Longer.” arXiv (2024).

The full code, trained checkpoints, and the 25‑species CDS dataset will be released under the OpenMed organization on Hugging Face.

Sources