basetenlabs/truss

The simplest way to serve AI/ML models in production

What it solves

Truss simplifies the process of deploying and serving AI/ML models in production. It removes the need for manual containerization, Dockerfile writing, and Kubernetes configuration, allowing developers to move from a model on Hugging Face to a production-ready API endpoint quickly.

How it works

Truss provides a CLI that packages model code, weights, and dependencies into a portable format. Users define their deployment specifications—such as hardware (GPUs), model weights, and inference engines—in a config.yaml file. The tool then handles the build process, including TensorRT-LLM optimization and containerization, and deploys the model to Baseten or other infrastructure.

Who it’s for

ML engineers and developers who want to deploy models from any Python framework (such as PyTorch, TensorFlow, vLLM, or transformers) without managing the underlying infrastructure or container orchestration.

Highlights

  • Framework Agnostic: Supports a wide range of frameworks including vLLM, SGLang, TensorRT-LLM, transformers, and diffusers.
  • Rapid Iteration: Features live reload and a "watch" mode to sync changes to deployed models without full rebuilds.
  • Simplified Configuration: Uses a YAML-based configuration for specifying hardware and model settings, with IDE support for autocompletion and validation.
  • Production Ready: Built-in support for GPU configuration, secrets, caching, and autoscaling.

Related

  • Project
    gpustack/gpustack

    An open-source GPU cluster manager for AI model serving and instance provisioning that orchestrates inference engines across hybrid environments.

  • Project
    scaleapi/llm-engine

    An open-source engine for fine-tuning and serving large language models, providing tools to deploy models on hosted or self-managed Kubernetes infrastructure.

  • Project
    Arm-Examples/ML-examples

    A collection of machine learning examples and tutorials for deploying and optimizing AI models on Arm Ethos NPUs and Cortex-based platforms.

  • Project
    vllm-project/llm-compressor

    A library for quantizing and compressing large language models (weights, activations, KV‑cache, etc.) so they run efficiently with the vLLM inference engine. Supports many algorithms (GPTQ, AWQ, AutoRound, REAP pruning, etc.) and formats (FP8, NVFP4, INT4, mixed‑precision). Provides one‑line API, DDP + disk‑offloading for multi‑terabyte models, and pre‑quantized checkpoints for popular LLMs.

  • Project
    georgian-io/LLM-Finetuning-Toolkit

    A config-based CLI tool for launching and managing LLM fine-tuning experiments, allowing users to run ablation studies across different models, prompts, and optimization techniques.