BLOOM: The World's Largest Open Multilingual Language Model

Hugging Face and the BigScience project have released BLOOM, a 176-billion parameter open-access multilingual language model. This release aims to democratize access to large language models (LLMs) by providing a transparently trained model that academia, nonprofits, and smaller research labs can study and build upon.

Model Scale and Multilingual Capabilities

BLOOM is designed to support a wide array of languages, making it the first model with over 100 billion parameters to support many of the languages it covers. It features 176 billion parameters and can generate text in 46 natural languages and 13 programming languages, including Spanish, French, and Arabic.

Collaborative Development and Training

BLOOM is the result of a year-long collaboration involving over 1,000 researchers from more than 70 countries and 250+ institutions. The final training run took place over 117 days between March 11 and July 6, 2022, on the Jean Zay supercomputer in France. This effort was supported by a compute grant estimated at ⁀3M from the French research agencies CNRS and GENCI.

Open Access and Responsible AI Licensing

To ensure the model is accessible for research and study, BLOOM is available for download and execution on local machines or cloud providers. Access is governed by the Responsible AI License (RAIL), which was developed during the BigScience project.

To further transparency, the project has released the intermediary checkpoints and optimizer states of the training process. For users without high-end hardware (such as 8 A100 GPUs), an inference API backed by Google’s TPU cloud and a FLAX version of the model is available for prototyping and low-scale use.

Future Development Roadmap

BLOOM is intended as a starting point for a family of models rather than a standalone release. Planned improvements and developments include:

  • Making the model instructable, following the work done with T0++.
  • Adding support for more languages.
  • Compressing the model into more usable versions while maintaining performance levels.
  • Using BLOOM as a foundation for more complex architectures.

Sources