ashhart/TensorFold

LLM Inference Engine for Metal, CUDA and Vulkan.

What it solves

TensorFold provides a high-performance, standalone way to serve language models on Apple Silicon and NVIDIA GPUs without requiring Python or MLX installations. It simplifies the deployment of specific qualified models by providing a native binary that handles checkpoint reading, tokenization, and kernel execution directly.

How it works

Built with Zig, the engine runs Metal or CUDA kernels directly to execute model weights. It supports an OpenAI-compatible API for chat completions and Anthropic Messages. Key technical features include:

  • Living Weights: An experimental feature that allows a running model to learn new facts by writing them directly into the checkpoint files, making the learned information persistent across restarts.
  • Context Compaction: Instead of refusing requests that overflow the context window, it can summarize older conversation turns into a "memory note" while keeping recent turns verbatim.
  • Exact Decoding: Ensures that drafted responses match the reference output of the native engine exactly.
  • Prompt Caching: Supports retained-prefix budgets to speed up processing of common prompt prefixes.

Who it’s for

Developers and users who want to run LLMs locally on macOS (M1-M5) or Linux (NVIDIA Ampere/Blackwell) with minimal dependencies and high efficiency.

Highlights

  • Zero Python Dependency: Serves models as a native Zig binary.
  • Living Weights: Enables real-time, persistent learning of new facts into model weights.
  • ** uma 3D Visualization**: Includes a 3D view to monitor how facts are integrated into neurons and layers.
  • Broad Hardware Support: Optimized for Apple Silicon (Metal) and NVIDIA GPUs (CUDA).
  • Context Management: Intelligent compaction to prevent context overflow.

Related

  • Project
    cubist38/mlx-openai-server

    An OpenAI-compatible API server for running local MLX models on Apple Silicon, supporting text, multimodal, image, and audio models.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    tensorflow/serving

    A high-performance serving system for machine learning models designed to manage model lifetimes and provide versioned inference via gRPC and HTTP in production environments.

  • Project
    triton-inference-server/server

    An open-source inference serving software that streamlines the deployment of AI models from multiple frameworks across cloud, data center, edge, and embedded devices.

  • Project
    InfiniTensor/InfiniTensor

    InfiniTensor is an open‑source C++ runtime that imports ONNX graphs, applies generic graph rewrites, and executes them on CPU or on custom accelerator back‑ends (InfiniOps/InfiniRT). It offers Python bindings, a test suite, and a modular design that lets hardware vendors plug in their own kernels.