OpenAI Announces Efficient Training of Language Models for Fill-in-the-Middle

TL;DR

OpenAI introduced a data‑augmentation technique that lets standard autoregressive language models learn to fill in missing spans (fill‑in‑the‑middle, FIM) while preserving their original left‑to‑right generation performance, making FIM a practical default training objective.

Why FIM Matters

Training models to perform fill‑in‑the‑middle expands their utility for editing, code completion, and interactive writing, where users often need to insert or replace text rather than generate from scratch. By integrating FIM during pre‑training, OpenAI shows that models can acquire this capability without additional fine‑tuning or architectural changes.

Core Technique: Span‑Shift Transformation

The method modifies the training corpus by selecting a contiguous span from the middle of a document and moving it to the end of the same document. The model then sees the shifted text as a target to predict given the surrounding context. This simple transformation creates an infilling task while keeping the original left‑to‑right ordering intact for the unshifted portion.

Empirical Findings

  • No Degradation in Generative Ability: Across multiple model scales, perplexity on standard left‑to‑right language modeling benchmarks remains unchanged after extensive FIM training.
  • Strong Infill Performance: Models trained with a high proportion of transformed examples achieve competitive results on dedicated infilling benchmarks released by OpenAI.
  • Scalability: The approach works consistently from small to large models, indicating that the transformation does not interfere with scaling laws.

Ablation Studies and Best‑Practice Settings

OpenAI conducted systematic ablations on three key hyperparameters:

  1. Transformation Frequency – The proportion of training examples that undergo the span‑shift. Results suggest using a moderate frequency (e.g., 30‑50%) balances infill skill and generative fidelity.
  2. Span Structure – Varying span length and position shows that randomizing both yields the most robust performance.
  3. Span Selection Method – Random selection outperforms heuristic or length‑biased strategies, simplifying implementation.

From these experiments, OpenAI recommends the following defaults:

  • Apply the transformation to roughly half of the training data.
  • Randomly choose span length uniformly within a predefined range (e.g., 10‑50 tokens).
  • Randomly locate the span within the document, avoiding extreme start or end positions.

Released Resources

  • Best‑in‑class FIM Model: OpenAI has deployed the top-performing infilling model via its API, allowing developers to query the model for fill‑in‑the‑middle predictions directly.
  • Infill Benchmarks: A suite of evaluation datasets and scripts has been open‑sourced to facilitate reproducible research on FIM capabilities.

Implications for Future Language Model Development

  • Default Training Objective: Given the negligible impact on left‑to‑right performance, OpenAI suggests that future autoregressive models adopt FIM training as a standard practice.
  • Efficiency Gains: The transformation incurs minimal computational overhead, making it attractive for large‑scale pre‑training pipelines.
  • Broader Applicability: The technique is model‑agnostic and can be applied to any autoregressive architecture, potentially accelerating the adoption of infilling features across the field.

"While this data augmentation has garnered much interest in recent years, we provide extensive evidence that training models with a large fraction of data transformed in this way does not harm the original left-to-right generative capability..." – OpenAI authors

Conclusion

OpenAI’s efficient FIM training method demonstrates that a simple dataset transformation can endow autoregressive language models with robust infilling abilities without compromising their core generative strengths. The released model, benchmarks, and recommended hyperparameters provide a clear pathway for the community to integrate fill‑in‑the‑middle functionality into future language models.

Sources