floneum/kalosm

Instant, controllable, local pre-trained AI models in Rust

What it solves

Kalosm is a Rust-based ecosystem designed to simplify the development of applications that integrate local or remote AI models. It provides a unified interface for interacting with various pre-trained models across different modalities (text, audio, and image) and includes tools for structured data generation and context management for LLMs.

How it works

The project consists of two primary components:

  • Kalosm: A high-level interface for pre-trained models. It supports models like Llama, Mistral, Phi, Whisper, Segment Anything, and Bert. It includes utilities for extracting context from various file formats (PDF, HTML, DOCX, etc.), chunking text, and integrating with vector databases for semantic search.
  • Fusor: A WGPU-based runtime for quantized ML inference. It uses the GGUF file format and leverages a kernel fusion compiler to optimize operation chains into single kernels, allowing models to run natively on various accelerators (Nvidia, AMD, Metal) or directly in the browser via WebGPU.

Who it’s for

Rust developers who want to build AI-powered applications with a consistent API, support for local quantized models, and the efficient execution of models on diverse hardware accelerators.

Highlights

  • Multimodal Support: Integrated support for text (LLMs), audio (Whisper), and image (Segment Anything) models.
  • Structured Generation: Uses a custom parser engine and sampler to constrain model outputs to specific Rust types, JSON, HTML, or XML via grammar-based constraints.
  • ** Firestore Integration**: Utilities for context extraction, web crawling, and scraping to feed LLM context.
  • Hardware Agnostic: Through Fusor, it targets multiple GPU architectures and the browser using WGPU.
  • Quantized Inference: Native support for quantized models to reduce memory and compute requirements.

Related

  • Project
    Abraxas-365/langchain-rust

    A Rust implementation of the LangChain framework for building composable LLM applications with support for various models, vector stores, and agents.

  • Project
    lucasjinreal/Crane

    Crane is a Rust‑based inference engine built on the Candle tensor library. It provides fast, cross‑platform (CPU, CUDA, Metal, experimental ROCm) execution of LLMs and multimodal models, offers an OpenAI‑compatible server, and includes ready‑to‑run examples for chat, OCR, ASR, TTS, and music transcription.

  • Project
    EricLBuehler/mistral.rs

    A fast, flexible LLM inference engine written in Rust that supports multimodal models, automatic quantization, and a built-in agentic runtime for tool and code execution.

  • Project
    jamjamjon/usls

    A cross-platform Rust library powered by ONNX Runtime for efficient inference of vision and vision-language models under 1B parameters.

  • Project
    foldl/chatllm.cpp

    A C++ LLM inference engine based on ggml that enables memory-efficient local execution of multimodal models with RAG and quantization support.