kaito-project/aikit

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

What it solves

AIKit is a platform designed to simplify the process of hosting, deploying, building, and fine-tuning large language models (LLMs). It removes the complexity of environment setup by providing a containerized approach to AI model management, allowing users to run models locally or in the cloud without needing complex toolchains.

How it works

AIKit leverages Docker and Podman to deliver models as container images. It uses LocalAI as its inference engine to provide an OpenAI-compatible REST API, making it a drop-in replacement for existing OpenAI clients. For fine-tuning, it integrates with Unsloth for memory-efficient training. It also supports OCI packaging, allowing models to be distributed as OCI artifacts using the CNCF ModelPack specification.

Who it’s for

Developers and AI engineers who want to quickly deploy open LLMs, fine-tune them for specific tasks, and distribute them via container registries, especially those working in air-gapped or edge environments.

Highlights

  • OpenAI Compatibility: Provides a REST API that works with any OpenAI-compatible client.
  • Hardware Flexibility: Supports AMD64 and ARM64 CPUs, NVIDIA CUDA, AMD ROCm, and Apple Silicon.
  • Container-First: Models are packaged as OCI artifacts, enabling easy distribution and deployment via Docker or Podman.
  • Fine-Tuning: Integrated support for Unsloth for fast and memory-efficient fine-tuning.
  • Security: Includes SBOMs, Provenance attestations, and signed images to ensure supply chain security.
  • Multi-modal Support: Capable of running both text-based LLMs and image generation models (e.g., Flux 1 Dev).