stochasticai/xTuring

What it solves

xTuring provides a streamlined way to fine-tune, evaluate, and run open-source Large Language Models (LLMs) privately. It addresses the complexity and high hardware costs typically associated with training models on custom data, allowing users to run these processes locally or within a private cloud (VPC) to maintain data privacy.

How it works

The project uses a high-level API to abstract the complexities of data preparation, training, and inference. It leverages efficiency techniques like LoRA (Low-Rank Adaptation) and low-precision quantization (INT8 and INT4) to reduce memory and compute requirements. It also integrates with the Intel® Extension for Transformers for optimized CPU inference. The system is built around a registry pattern, allowing users to create and manage various model variants (e.g., LLaMA, Qwen3, GPT-OSS) through a unified interface.

Who it’s for

It is designed for developers and organizations that need to personalize LLMs with their own data while keeping that data private, and those who want to run fine-tuning or inference on limited hardware, ranging from laptops to multi-GPU setups.

Highlights

• Broad Model Support: Supports a wide range of models including LLaMA 2, Qwen3, GPT-OSS, Mistral, and Mamba.
• Efficient Training: Implements LoRA and INT4/INT8 quantization to significantly cut GPU memory usage and training time.
• Private by Default: Designed for local or VPC deployment to ensure data privacy.
• Built-in Evaluation: Includes tools to evaluate model performance using metrics like perplexity.
• Interactive Tools: Provides a CLI for chatting with models and a Gradio-based UI playground for testing.
• Dataset Generation: Features utilities for bootstrapping instruction corpora using self-instruction and multimodal checkpoints like Qwen3-Omni.