xAI Grok Release Notes

Grok: Real-Time Knowledge and Wit

xAI has introduced Grok, an AI assistant designed to answer a wide range of questions with a rebellious streak and a sense of humor, modeled after the Hitchhiker’s Guide to the Galaxy. Grok distinguishes itself from other AI systems by integrating real-time knowledge of the world through the — — platform and a willingness to answer "spicy" questions that are typically rejected by other AI models.

Grok-1 Technical Performance

The engine powering the assistant is Grok-1, a frontier large language model (LLM) developed over four months. It follows a prototype model, Grok-0 (33B parameters), which approached LLaMA 2 (70B) capabilities while using half the training resources.

Benchmark Results

Grok-1 demonstrates strong reasoning and coding capabilities, surpassing models in its compute class such as GPT-3.5 and Inflection-1. Its performance on standard machine learning benchmarks is as follows:

Benchmark	Grok-1	GPT-3.5	LLaMa 2 70B	Inflection-1
GSM8k (8-shot)	62.9%	57.1%	56.8%	62.9%
MMLU (5-shot)	73.0%	70.0%	68.9%	72.7%
HumanEval (0-shot)	63.2%	48.1%	29.9%	35.4%
MATH (4-shot)	23.9%	23.5%	13.5%	16.0%

Real-World Validation

To mitigate the risk of data contamination from web-based benchmarks, xAI conducted a human-graded evaluation using the May 2023 Hungarian national high school finals in mathematics. Grok-1 achieved a score of 59% (Grade C), outperforming Claude-2 (55%) and trailing GPT-4 (68%).

Infrastructure and Engineering

To support the training and inference of Grok-1, xAI developed a custom stack utilizing Kubernetes, Rust, and JAX.

Rust was selected for the infrastructure layer to ensure high performance and reliability, reducing bugs in the distributed system and allowing a small team to maintain the system with minimal supervision. The engineering focus was placed on maximizing useful compute per watt and maintaining high Model Flop Utilization (MFU) despite the inherent unreliability of hardware at the scale of tens of thousands of GPUs.

Future Research Directions

xAI identifies reliable reasoning as the primary research goal to address the tendency of LLMs to generate false or contradictory information. The lab is pursuing five key research areas:

1. Scalable Oversight: Using AI to assist humans in verifying complex reasoning and code by looking up references and external tools.
2. Formal Verification: Integrating formal guarantees for code correctness to improve safety, reliability, and grounding.
3. Long-Context Understanding: Improving the discovery and retrieval of useful knowledge within specific contexts.
4. Adversarial Robustness: Reducing vulnerabilities that allow optimizers to exploit AI systems during training and serving.
5. Multimodal Capabilities: Adding vision and audio senses to enable real-time interactions.

Availability

As of November 3, 2023, Grok is in an early beta phase. A limited number of users in the United States can join a waitlist for early access to the prototype.