hardrave/NIGHTRUN

What it solves

NightRun is a local LLM runtime that allows a computer to boot directly into a chat interface without needing a conventional operating system (like Linux or Windows). It eliminates the overhead of a kernel, browser, or network stack, turning a machine into a dedicated, offline AI appliance.

How it works

Written in Rust, NightRun operates as a UEFI application. It leverages the machine's firmware to handle basic hardware interactions (like the display and keyboard) and loads a quantized model into RAM from a USB stick or SD card.

Key technical details include:

• Model Lifecycle: It uses a custom .nrm container format that verifies model data via CRC-32 checksums during the streaming process. Once loaded, storage is "sealed," meaning the system prevents further disk reads to ensure predictable performance and security.
• Inference Engine: It features hand-written quantized kernels (AVX2 for x86_64 and NEON for Raspberry Pi 5) to run models like Llama 3.2, Qwen3, and Granite 4.1 without needing dequantized copies.
• Validation: The project maintains strict token-for-token parity with llama.cpp to ensure correctness.

Who it’s for

It is designed for users who want a completely isolated, offline LLM experience on x86_64 PCs or Raspberry Pi 5 hardware, as well as systems programmers interested in bare-metal-style AI inference.

Highlights

• OS-less Boot: Boots directly from USB/SD card into an LLM prompt using UEFI Boot Services.
• Hardware Support: Compatible with 64-bit UEFI machines and Raspberry Pi 5.
• High Performance: Memory-bandwidth-bound decode speeds comparable to llama.cpp.
• Surgical Installer: A paranoid installer that prevents accidental flashing of system disks through strict device fingerprinting and manual confirmation.
• Custom Model Format: The .nrm format allows for zero-copy views of tensors and embedded tokenizers.