basetenlabs/truss

The simplest way to serve AI/ML models in production

What it solves

Truss simplifies the process of moving AI/ML models from development to production. It removes the need for developers to manually handle containerization, Dockerfile creation, Kubernetes configuration, and GPU setup, which are typically complex and time-consuming barriers to deploying scalable model APIs.

How it works

Truss provides a CLI that packages model serving logic, weights, and dependencies into a standardized format. Users can deploy models using a simple config.yaml file to specify hardware (like L4 GPUs) and the inference engine (such as TensorRT-LLM). It supports a wide range of frameworks including transformers, diffusers, PyTorch, and TensorFlow, as well as optimized engines like vLLM and SGLang. When pushed to a platform like Baseten, Truss handles the build process—including optional TensorRT optimization and quantization (e.g., FP8)—and turns the model into an OpenAI-compatible API endpoint.

Who it’s for

Machine learning engineers and developers who want to deploy their models to production without managing the underlying infrastructure or writing complex container orchestration scripts.

Highlights

  • Framework Agnostic: Supports almost all Python ML frameworks, including PyTorch, TensorFlow, and vLLM.
  • Zero-Config Containers: Automates containerization and dependency management so users can skip writing Dockerfiles.
  • Optimized Inference: Integrated support for TensorRT-LLM for high-performance serving.
  • Developer-Friendly Loop: Features live reload for faster iteration during development.
  • IDE Support: Includes a JSON schema for config.yaml to provide autocompletion and validation in modern editors.

Related

  • Project
    gpustack/gpustack

    An open-source GPU cluster manager that orchestrates inference engines and provisions GPU instances for scalable AI model serving.

  • Project
    scaleapi/llm-engine

    An open-source engine for fine-tuning and serving large language models, providing tools to deploy models on hosted or self-managed Kubernetes infrastructure.

  • Project
    Arm-Examples/ML-examples

    A collection of machine learning examples and tutorials for deploying and optimizing AI models on Arm Ethos NPUs and Cortex-based platforms.

  • Project
    georgian-io/LLM-Finetuning-Toolkit

    A config-based CLI tool for launching and managing LLM fine-tuning experiments, allowing users to run ablation studies across different models, prompts, and optimization techniques.

  • Project
    GoogleCloudPlatform/cluster-toolkit

    Google Cloud Cluster Toolkit is an open‑source CLI that turns YAML blueprints into Terraform/Packer code to provision AI/ML or HPC clusters (Slurm or GKE) on Google Cloud, and also lets you submit containerised AI jobs. It provides pre‑built bundles, automatic dependency handling, and advanced features like TPU slicing and Pathways support.