InfiniTensor/InfiniTensor

InfiniTensor – A lightweight, extensible runtime for AI compute graphs

What it is

• InfiniTensor is an open‑source C++ library that loads, transforms, and runs computation graphs (e.g. ONNX models). It ships a native CPU backend and can delegate work to custom accelerator back‑ends called InfiniOps and InfiniRT.
• The core library is hardware‑agnostic; the actual device‑specific kernels live in separate SDKs that you install alongside InfiniTensor.

Why it matters

• Modern AI workloads are often expressed as graph‑based intermediate representations (IRs) such as ONNX. A runtime that can import that IR, apply graph‑level optimisations, and then execute on either CPU or specialised accelerators is a key piece of the inference stack.
• By separating the graph engine (InfiniTensor) from the device kernels (InfiniOps/InfiniRT), the project lets hardware vendors plug in their own implementations without modifying the core runtime.

Key features (as described in the README)

• ONNX importer – parses ONNX files into an internal graph representation.
• Graph transformation layer – supports generic rewrites; future work mentions RefactorGraph and EinNet for more advanced, derivation‑based optimisations.
• CPU runtime – a built‑in, pure‑C++ execution engine (pyinfinitensor Python bindings) for development and fallback.
• Accelerator delegation – when an InfiniOps/InfiniRT build is present, operators are dispatched to the accelerator; unsupported ops raise explicit errors rather than silently falling back.
• Python API – installable via make install-python, exposing the runtime as the pyinfinitensor package.
• Testing suite – C++ unit tests, ONNX‑frontend tests, and Python API tests are provided.

How to get started

1. Clone the repo and initialise submodules (git submodule update --init --recursive).
2. Build and install the CPU‑only Python package:

   make install-python INFINI=OFF PYTHON=$(command -v python3)
   

3. For accelerator support, first install matching InfiniOps and InfiniRT SDKs, then rebuild with INFINI=ON and point to the SDK prefixes.
4. Use the provided make targets (build, test-cpp, test-onnx, test-api) to compile and verify the installation.

Roadmap highlights

• RefactorGraph – next‑gen graph framework (linked as a separate repo).
• EinNet – derivation‑based tensor program optimisation (experimental branch NNET_e2e).
• PET – partially equivalent transformations for automated correction of tensor programs.
• Ongoing expansion of accelerator support via the InfiniOps/InfiniRT ecosystem.

Who might use it

• Researchers building custom AI operators or exploring new graph optimisation passes.
• Hardware vendors who want a thin, reusable front‑end to expose their kernels to ONNX models.
• Developers needing a lightweight, Python‑friendly runtime for inference on CPUs with the option to plug in proprietary accelerators.

Where to learn more

• Installation guide, user guide, and distributed example are in the /docs and /examples directories.
• The accompanying research papers EinNet (OSDI 2023) and PET (OSDI 2021) provide deeper context on the optimisation techniques the project builds upon.