tairov/llama2.mojo

What it solves

llama2.mojo is a high-performance implementation of Llama 2 inference in the Mojo programming language. It addresses the need for efficient, hardware-optimized CPU inference for small-to-medium sized LLMs without relying on heavy frameworks, providing a significant speedup over Python implementations and competitive performance compared to C-based alternatives.

How it works

The project leverages Mojo's SIMD (Single Instruction, Multiple Data) and vectorization primitives to optimize matrix multiplications and other tensor operations. It utilizes a persistent worker pool for multi-threaded inference, allowing it to outperform llama2.c and llama.cpp on specific baby-llama models on CPU.

Who it’s for

Developers and researchers interested in exploring the efficiency of the Mojo language for AI workloads, as well as those looking for a lightweight, pure-Mojo implementation for running Llama-style models on CPU.

Highlights

• High Performance: Boosts Python performance by nearly 250x and outperforms llama2.c by 30% in multi-threaded inference.
• Hardware Optimization: Specifically optimized for CPU inference using Mojo's advanced hardware-level features.
• Model Support: Supports various "baby Llama" models (stories 260K to 110M) and TinyLlama-1.1B-Chat-v0.2.
• Single File: Implements Llama 2 inference in a single file of pure Mojo.