knights-analytics/hugot

Onnx transformer pipelines in Golang

What it is

Hugot is a Go library that lets you run Hugging Face‑style transformer pipelines directly from Go code. It works with models exported to the ONNX format and supports a wide range of tasks – text classification, question answering, image segmentation, audio transcription, zero‑shot classification, and more. The library provides three interchangeable back‑ends:

  • Pure‑Go (no C dependencies, good for small models)
  • ONNX Runtime (ORT) – the fastest CPU/GPU backend
  • OpenXLA – enables TPU support and other XLA accelerators.

Who might use it

  • Go developers who want to embed AI inference in services without calling a separate Python server.
  • ML engineers deploying Hugging Face models in Go‑based production stacks.
  • Teams that need to run inference on‑premise, on CPUs, GPUs, or TPUs, and prefer a single‑language code base.

Core features (as described in the README)

Feature Details
Model compatibility Any Hugging Face model that can be exported to ONNX (via Optimum) works, and predictions match the Python reference.
Supported pipelines Over 30 pipelines covering text, vision, audio, multimodal and classic ML (e.g., textClassification, imageSegmentation, automaticSpeechRecognition, zeroShotClassification, objectDetection, etc.). Some pipelines are ORT‑only.
Back‑end selection Choose at compile‑time with build tags (-tags ORT, -tags XLA, -tags ALL). The Go backend needs no cgo; ORT and XLA give hardware acceleration.
Hardware acceleration CPU (default), NVIDIA GPU via CUDA (ORT or XLA), TPU (XLA only), plus support for TensorRT, DirectML, CoreML, OpenVINO through ONNX Runtime.
Training / fine‑tuning Uses goMLX to load an ONNX model, fine‑tune it, and write it back to ONNX. Currently available for the FeatureExtraction pipeline (useful for semantic‑search embeddings).
Generative AI Runs generative models through ONNX Runtime’s Generative AI engine; supports concurrent requests and batching via WithGenerativeEngine.
Installation go get github.com/knights-analytics/hugot; optional C libraries for ORT/XLA can be pulled from releases or Docker image.
Example usage The README includes a full Go program that downloads a sentiment model, creates a text‑classification pipeline, runs it on a batch of strings, and prints JSON results.

How it works (high‑level)

  1. Model download – hugot.DownloadModel fetches an ONNX model from the Hugging Face hub.
  2. Session creation – NewGoSession, NewXLASession, or NewORTSession creates a runtime context bound to a chosen backend.
  3. Pipeline construction – Pass a config (e.g., TextClassificationConfig) to session.NewPipeline.
  4. Inference – Call pipeline.RunPipeline(ctx, inputs); the library handles tokenization, model execution, and output parsing.
  5. Training – Create a Trainer from a session, supply a dataset (JSONL with sentence1, sentence2, score), and call trainer.Train. The fine‑tuned model can be saved and re‑used as any other pipeline.

Ecosystem & licensing

  • The project is open‑source (MIT‑style badge links suggest a permissive license).
  • It builds on goMLX for the pure‑Go and XLA back‑ends and on ONNX Runtime for the ORT back‑end.
  • Docker images are provided for quick testing with all dependencies pre‑installed.

When to pick Hugot

  • You already write services in Go and want to avoid a Python‑to‑Go RPC layer.
  • You need low‑latency, on‑device inference (e.g., edge services, micro‑services, or real‑time APIs).
  • Your models are exportable to ONNX (most Hugging Face models are).
  • You want optional GPU/TPU acceleration without leaving the Go ecosystem.

All information above is taken directly from the repository’s README; no external assumptions have been added.

Related

  • Project
    yalue/onnxruntime_go

    onnxruntime_go is a Go wrapper for Microsoft's ONNX Runtime that enables loading and running ONNX neural networks from Go code. It manually loads the ONNX Runtime shared library at runtime, providing cross-platform support including Windows, and uses Go generics to simplify tensor handling.

  • Dispatch
    Hugging Face Accelerated Inference API Optimization

    Hugging Face achieved a 100x speedup in transformer inference for its Accelerated Inference API by combining high-level library optimizations, Rust-based tokenization, and hardware-specific compilation.

  • Dispatch
    Convert Transformers to ONNX with Hugging Face Optimum

    Hugging Face provides three methods to convert Transformers models to ONNX, ranging from the low-level torch.onnx API to the high-level Optimum library.

  • Dispatch
    Accelerating Hugging Face Models with ONNX Runtime

    Hugging Face and ONNX Runtime enable performance acceleration for over 130,000 models, including a latency reduction of up to 74.30% for the whisper-tiny model compared to PyTorch.

  • Project
    gomlx/gomlx

    A full-featured machine learning and math framework for Go, similar to PyTorch or Jax, supporting training, fine-tuning, and inference across multiple backends including XLA and ONNX.