xAI Grok-1 Open Release

xAI has released the base model weights and network architecture of Grok-1, a 314 billion parameter Mixture-of-Experts (MoE) model. This release provides the raw base model checkpoint from the pre-training phase, enabling developers and researchers to build upon the model's foundational capabilities under the Apache 2.0 license.

Grok-1 Model Architecture and Scale

Grok-1 is a large-scale Mixture-of-Experts model featuring 314 billion parameters. To optimize computational efficiency during inference, the model is designed so that only 25% of its weights are active for any given token.

Training Specifications and Timeline

xAI trained Grok-1 from scratch using a custom training stack built on JAX and Rust. The pre-training phase for this specific checkpoint concluded in October 2023.

Model State and Licensing

Because this is a raw base model checkpoint, Grok-1 is not fine-tuned for specific applications, such as dialogue or instruction following. It represents the foundational state of the model after pre-training on a large volume of text data. The weights and architecture are released under the Apache 2.0 license, allowing for broad use and modification.

Implementation and Access

Technical instructions for deploying and using the Grok-1 model are available via the official xAI GitHub repository at github.com/xai-org/grok.