kaito-project/aikit

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

What it solves

AIKit provides a unified platform to simplify the hosting, deployment, and fine-tuning of large language models (LLMs). It removes the complexity of managing infrastructure, GPU drivers, and model formats by providing pre-made, containerized model images that can be run with a single command.

How it works

AIKit leverages containerization (Docker/Podman) to package models and their dependencies. It uses LocalAI as an inference engine to provide an OpenAI-compatible REST API, allowing users to integrate with existing AI clients. For fine-tuning, it integrates with Unsloth for memory-efficient training. It also supports OCI packaging, allowing models to be distributed as OCI artifacts following the CNCF ModelPack specification.

Who it’s for

Developers and organizations looking to deploy open LLMs locally or on Kubernetes without needing deep expertise in ML infrastructure, as well as those requiring air-gapped environments or high supply chain security for their AI models.

Highlights

  • OpenAI API Compatibility: Drop-in replacement for OpenAI clients using a REST API.
  • Hardware Flexibility: Supports AMD64 and ARM64 CPUs, NVIDIA CUDA, AMD ROCm, and Apple Silicon.
  • Container-First Approach: Deploy models via Docker or Podman with minimal image sizes using chiseled images.
  • Fine-Tuning & Packaging: Built-in support for Unsloth fine-tuning and OCI-compliant model distribution.
  • Enterprise Ready: Kubernetes deployment ready with support for SBOMs and signed images for supply chain security.

Related

  • Project
    kitops-ml/kitops

    A CNCF open-source tool for packaging, versioning, and securely sharing AI/ML projects using OCI-compliant artifacts called ModelKits.

  • Project
    mudler/LocalAI

    An open-source AI engine that lets you run LLMs, vision, voice, and image models locally on any hardware with drop-in OpenAI-compatible APIs.

  • Project
    gpustack/gpustack

    An open-source GPU cluster manager that orchestrates inference engines and provisions GPU instances for scalable AI model serving.

  • Project
    mlc-ai/mlc-llm

    MLC LLM is a universal deployment engine and ML compiler that enables high-performance native execution of large language models across diverse hardware and platforms.

  • Project
    open-webui/open-webui

    Open WebUI is an open‑source, self‑hosted web interface for LLMs that works with any OpenAI‑compatible API and with local Ollama models. It provides a rich UI (chat, notes, channels, calendar, voice/video, image generation), extensible plugin system, RBAC, multi‑model support, built‑in RAG (vector DBs, web search), an “Open Terminal” for agents to run code, and enterprise features like LDAP/SSO, SCIM provisioning, and observability. Install via pip, Docker (CPU/GPU, bundled Ollama), or Kubernetes. The ecosystem includes a desktop app, a mobile‑first coding agent, and tools for syncing external knowledge bases.