Rusty Llama Webapp – Quick Overview

What it is – A minimal chatbot web application written entirely in Rust. It uses:

• Leptos (a Rust‑based UI framework) for the front‑end, styled with TailwindCSS.
• Actix as the HTTP server.
• mistral.rs to run inference of open‑source LLMs (e.g., Qwen‑3.8 B) directly in the same process.

Why it matters – Shows that a full‑stack AI‑powered app can be built without any Python, Node, or external inference service. The model runs locally on the same machine, using either Apple Metal, CUDA, or CPU‑only back‑ends.

---

How to get it running

1. Install the right Rust toolchain

   rustup target add wasm32-unknown-unknown
   cargo install cargo-leptos --locked   # provides `cargo leptos`
   

   Requires Rust 1.88+.

2. Set up GPU support

   • Apple Silicon – keep the metal feature in Cargo.toml. Install the Metal toolchain:

     xcodebuild -runFirstLaunch
     xcodebuild -downloadComponent MetalToolchain
     xcrun metal --version
     

   • NVIDIA – replace metal with cuda in the mistralrs dependency.
   • CPU‑only – simply remove the feature flag.

3. Configure the model (defaults to Qwen‑3.8 27B, 4‑bit quantized, ~17 GB)

   • Create a .env file (or edit the existing one) with:

     MODEL_REPO=bartowski/Qwen3.8-27B-GGUF
     MODEL_FILE=Qwen3.8-27B-Q4_K_M.gguf
     

   • On first start, mistral.rs will download the GGUF file into ~/.cache/huggingface.
   • To use another model, change MODEL_FILE or point MODEL_PATH at a local .gguf file or shard directory.
   • If the model lacks an embedded chat template, set CHAT_TEMPLATE to a Jinja string or a JSON file.

4. Run the app

   cargo leptos watch   # builds the Rust server, compiles Tailwind, and watches for changes
   

   Then open http://localhost:3000/ in a browser.

   The first build may take several minutes because mistral.rs compiles Metal/CUDA kernels.

---

What you’ll see

• A simple chat UI (see the GIF in the repo) where you can type prompts and get responses from the locally‑running LLM.
• The backend performs inference in‑process, so latency is comparable to running the model directly via a Python library, but you stay entirely within the Rust ecosystem.

---

Who might use this

• Rust developers who want to experiment with LLM inference without learning Python.
• Edge‑device hobbyists (Apple Silicon or NVIDIA GPUs) looking for a self‑contained AI chatbot.
• Educators demonstrating end‑to‑end AI pipelines in a single language.

---

Limitations & notes

• Only GGUF‑format models are supported; older GGML .bin files won’t work.
• The default model is 27 B parameters; you’ll need enough RAM/VRAM (≈17 GB for the 4‑bit version).
• GPU acceleration depends on the correct feature flag and a working Metal/CUDA toolchain.

---

Bottom line – Rusty Llama is a practical, minimal example of building a web‑based chatbot in pure Rust, leveraging mistral.rs for on‑device LLM inference and Leptos for a reactive UI.