zhudotexe/kani

What it solves

Kani is a lightweight, non-opinionated framework designed to simplify the creation of chat-based language model applications. It removes the boilerplate associated with interfacing with various LLM providers while avoiding the "hidden prompt hacks" and heavyweight complexity found in larger frameworks like LangChain.

How it works

Kani provides a model-agnostic interface that abstracts the interaction with different backends (OpenAI, Anthropic, Google, Hugging Face, llama.cpp, and vLLM). It manages chat state, token counting, and memory management automatically. For tool usage, it uses a Python decorator (@ai_function) to expose local functions to the model, handling validation and retries for hallucinated parameters.

Who it’s for

It is designed for NLP researchers, hobbyists, and developers who want fine-grained control over their prompts and control flow without the bloat of opinionated frameworks.

Highlights

• Model Agnostic: Easily switch between different LLM backends with minimal code changes.
• Function Calling: Simple integration of Python functions as tools with built-in error feedback and retry logic.
• Asynchronous Design: Built from the ground up to support parallel chat sessions.
• Multimodal Support: Integrated handling of images, audio, and video inputs via extensions.
• CLI Tool: A zero-setup command-line interface for rapid LLM testing.