tenstorrent/tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.

What it solves

TT-Metal provides a software stack for Tenstorrent hardware, enabling developers to run and optimize neural networks on specialized AI chips. It bridges the gap between high-level model development and low-level hardware execution, allowing for efficient deployment of large-scale models like Llama 3 and Qwen.

How it works

The project consists of two primary layers:

  • TT-NN: A Python and C++ library that provides high-level neural network operations, allowing users to define and run models.
  • TT-Metalium: A low-level programming model that allows developers to write custom kernels directly for Tenstorrent hardware, providing fine-grained control over compute and data movement.

The stack includes extensive tooling for profiling, debugging, and visualization, ensuring developers can monitor performance and troubleshoot hardware-level execution.

Who it’s for

This project is for AI researchers, hardware engineers, and systems developers working with Tenstorrent hardware who need to optimize model performance, implement custom kernels, or debug neural network execution at the hardware level.

Highlights

  • High-Performance Libraries: Includes both high-level (TT-NN) and low-level (TT-Metalium) programming interfaces.
  • Model Support: Optimized support for major LLMs like Llama 3.3 and Qwen 2.5, as well as audio models like Whisper.
  • Comprehensive Tooling: Features a suite of debugging and analysis tools, including the Tracy Profiler, DPRINT for kernel debugging, and the TT-NN Visualizer.
  • Hardware-Specific Optimization: Provides detailed technical reports on matrix multiplication, memory management, and multi-chip scaling to maximize hardware utilization.

Related

  • Project
    tenstorrent/tt-forge

    An open-source AI compiler stack for Tenstorrent hardware that enables the execution and training of models from PyTorch, JAX, and ONNX.

  • Project
    tenstorrent/tt-mlir

    An open-source compiler framework based on MLIR that optimizes and compiles machine learning models for Tenstorrent's custom AI hardware accelerators.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    alibaba/MNN

    A lightweight deep learning framework optimized for high-performance inference and training on mobile, IoT, and embedded devices.

  • Project
    NVIDIA/TensorRT

    A high-performance inference optimizer and runtime from NVIDIA that accelerates AI model deployment and throughput on NVIDIA GPUs.