floneum/kalosm

Instant, controllable, local pre-trained AI models in Rust

What it solves

Kalosm provides a simplified, unified interface for integrating pre-trained AI models into Rust applications. It removes the complexity of managing different model architectures and hardware acceleration, allowing developers to easily deploy local or remote models for text, audio, and image processing.

How it works

The project consists of two primary components:

  1. Kalosm: A high-level interface that supports a variety of pre-trained models (such as Llama, Mistral, Phi, Whisper, and BERT) across different modalities. It includes utilities for context extraction from various file formats, web crawling, and structured generation using a custom parser engine that constrains model outputs to specific Rust types or grammars.
  2. Fusor: A compiler-backed runtime for quantized ML inference. It uses an e-graph compiler to fuse operation chains into optimized kernels for CPUs and WebGPU accelerators, enabling efficient local inference without requiring manual shader code.

Who it’s for

Rust developers who want to build AI-powered applications with local or remote models without needing to deep expertise in ML own hardware-specific optimization.

Highlights

  • Multi-modal support: Integrated support for LLMs, audio transcription (Whisper), and image segmentation (SAM).
  • Structured Generation: Ability to constrain model outputs to arbitrary Rust types using #[derive(Parse, Schema)] or custom grammars.
  • JSON/HTML/XML support: Fast, structure-aware acceleration for generating complex data formats.
  • Hardware Acceleration: Local inference optimized for CPUs and WebGPU via the Fusor runtime.
  • RAG Utilities: Built-in tools for extracting, chunking, and retrieving context from PDF, HTML, and other documents.

Related

  • Project
    Abraxas-365/langchain-rust

    A Rust implementation of the LangChain framework for building composable LLM applications with support for various models, vector stores, and agents.

  • Project
    lucasjinreal/Crane

    Crane is a Rust‑based inference engine built on the Candle ML library. It loads GGUF models (LLMs, vision‑language, OCR, ASR, TTS, etc.) and runs them fast on CPU, CUDA, Apple Metal, or AMD ROCm, providing an OpenAI‑compatible server and easy‑to‑use Rust APIs.

  • Project
    EricLBuehler/mistral.rs

    A fast, flexible LLM inference engine written in Rust that supports multimodal models, agentic tool execution, and OpenAI/Anthropic compatible APIs.

  • Project
    jamjamjon/usls

    A cross-platform Rust library powered by ONNX Runtime for efficient inference of vision and vision-language models under 1B parameters.

  • Project
    foldl/chatllm.cpp

    A pure C++ LLM inference engine based on ggml that enables real-time multimodal chat with RAG and quantization for efficient local execution on CPU and GPU.