roboflow/inference

Turn any computer or edge device into a command center for your computer vision projects.

roboflow/inference – Self‑hosted computer‑vision inference server

What it is

  • A Python package (inference) that runs a local (or Docker‑based) server turning any computer, edge device, or GPU machine into a full‑featured computer‑vision hub.
  • It lets you load your own fine‑tuned models or use a catalog of pre‑trained foundation models (e.g., Florence‑2, CLIP, SAM‑2).
  • The core idea is Workflows: composable blocks that chain model inference, traditional CV steps (OCR, barcode reading, template matching), and custom Python code into reusable pipelines.

Key capabilities

  • Model serving – host any model you’ve trained with Roboflow or pull public models from the catalog.
  • Video‑stream processing – ingest RTSP, webcam, or other streams, with GPU‑accelerated batching, decoding, and multiprocessing.
  • Workflow engine – drag‑and‑drop style blocks for detection, segmentation, tracking, counting, measuring, notifications, and more. You can also write your own Python blocks.
  • API & SDK – a REST API (OpenAPI docs at /docs) and a Python SDK (inference-sdk) for programmatic control, including starting/stopping video pipelines and consuming results.
  • Hybrid cloud/edge – run entirely on‑prem for free (no credit usage) or connect to Roboflow Cloud for private models, dataset upload, model monitoring, and managed deployments.
  • Enterprise extensions – optional source‑available enterprise module for rugged Jetson‑based “Flowbox” devices, OPC/MQTT integration, and device management.

Quick start

pip install inference-cli
inference server start --dev   # pulls the right Docker image and launches a Jupyter notebook at http://localhost:9001/notebook/start

The notebook walks you through loading a model, creating a workflow, and connecting a camera.

Typical use‑cases

  • Retail checkout: detect items, compute totals, push results to a database.
  • Smart parking: license‑plate detection, vehicle tracking, Telegram alerts.
  • Quality‑control on a production line: combine object detection with OCR/barcode checks.
  • Visual search or content moderation: run CLIP or SAM‑2 on live streams to find frames matching a text prompt.

Installation & platform support

  • Distributed as a pip package; the server runs inside Docker (CPU or GPU images).
  • Supports Linux, Windows, macOS, NVIDIA Jetson, Raspberry Pi, and generic cloud VMs.
  • GPU acceleration works via NVIDIA Container Toolkit.

Licensing

  • The core inference code is Apache 2.0.
  • Individual models keep their original licenses (e.g., AGPL‑3.0 models must be used under that license).
  • Enterprise‑only features are source‑available under a separate commercial license.

Where to learn more

Bottom line Roboflow Inference is a production‑ready, self‑hostable platform for building, chaining, and deploying computer‑vision models at scale, with a strong focus on video streams and easy integration into existing systems.

Related

  • Project
    nndeploy/nndeploy

    A high-performance AI deployment framework that uses visual workflows to deploy AI models across desktop, mobile, and edge devices.

  • Project
    triton-inference-server/server

    An open-source inference serving software that streamlines the deployment of AI models from multiple frameworks across cloud, data center, edge, and embedded devices.

  • Project
    InfiniTensor/InfiniTensor

    InfiniTensor is an open‑source C++ runtime that imports ONNX graphs, applies generic graph rewrites, and executes them on CPU or on custom accelerator back‑ends (InfiniOps/InfiniRT). It offers Python bindings, a test suite, and a modular design that lets hardware vendors plug in their own kernels.

  • Project
    hyperspaceai/hyperspace-node

    A decentralized P2P AI inference network that allows users to contribute GPU, CPU, and storage resources to run AI models and earn rewards.

  • Project
    leejet/stable-diffusion.cpp

    stable‑diffusion.cpp is a pure C/C++ inference engine for diffusion‑based image (and some video) models. Built on ggml, it runs on CPU, CUDA, Vulkan, Metal, OpenCL, and SYCL, supports dozens of models (Stable Diffusion, FLUX, Qwen‑Image, etc.), multiple weight formats, and extensions like LoRA, ControlNet, and upscaling. It offers pre‑built binaries, a one‑line CLI, and a rich ecosystem of language bindings and UI front‑ends.