nnstreamer/nnstreamer

NNStreamer – Neural‑Network‑Powered Media Pipelines

What it is

• A collection of GStreamer plugins that let you treat a neural‑network model as just another media filter. In practice you can build a streaming pipeline (video, audio, sensor data, etc.) and insert inference steps anywhere in the flow.

Why it matters

• GStreamer is the de‑facto standard framework for handling multimedia streams on Linux, Android, Tizen, macOS and more. By plugging AI models into that ecosystem, developers get:
  • Low‑latency, on‑device inference – the data never has to leave the pipeline.
  • Modular composition – combine multiple models (e.g., object detection → pose estimation) in a single graph.
  • Hardware‑accelerated execution – sub‑plugins wrap a wide range of accelerators (TensorRT, Edge‑TPU, Movidius, ARM‑NN, Qualcomm QNN, etc.).

Key features (as described in the README)

• Supports major frameworks: TensorFlow, TensorFlow‑Lite, PyTorch, Caffe, ONNX, TVM, etc.
• Provides composite and multi‑modal pipelines, allowing several input sources and parallel model branches.
• Official binary packages for many platforms (Ubuntu, Tizen, Android, Yocto, macOS) and APIs in C/C#, Java, and plain C.
• Continuous‑integration tested builds with coverage, static analysis and daily releases.
• A growing list of example applications (pose estimation, image classification, object detection, face‑landmark tracking, etc.) and documented edge‑AI use‑cases.

Typical use‑case workflow

1. Choose a model (e.g., a TensorFlow‑Lite pose‑estimation model).
2. Create a GStreamer pipeline using the nnstreamer elements, e.g.

   gst-launch-1.0 filesrc location=video.mp4 ! decodebin ! videoconvert ! \
   tensor_filter framework=tflite model=pose.tflite ! \
   tensor_decoder mode=pose ! videoconvert ! autovideosink
   

3. The tensor_filter element runs inference on each frame, optionally on a supported accelerator, and passes the results downstream.
4. Deploy the same pipeline on‑device (Android, Tizen, embedded Linux) with minimal code changes.

Who should look at this

• Media‑engineers who want to add AI without rewriting their existing GStreamer‑based code.
• Embedded/edge‑AI developers needing a unified way to run inference on heterogeneous hardware.
• Researchers prototyping multi‑model pipelines (e.g., detection → classification → tracking) in a streaming fashion.

Where to start

• Follow the Getting Started links for your OS (Linux, macOS, Android).
• Browse the nnstreamer-example repository for ready‑made demos.
• Check the architectural wiki for deeper design details.

---

All information above is taken directly from the repository’s README; no additional features have been inferred.