microsoft/psi

Platform for Situated Intelligence

Platform for Situated Intelligence (\psi)

What it is – An open‑source, .NET‑based framework for building multimodal, real‑time AI systems. It lets you stream, synchronize, process, visualise and persist data from many sensors (audio, video, depth, etc.) and combine those streams with AI components while respecting latency constraints.

Key capabilities

  • High‑performance streaming infrastructure – core concept is a stream that can be transformed, merged, or replayed.
  • Tooling – PsiStudio for visualising and annotating streams; utilities for data capture, debugging and export.
  • Component ecosystem – ready‑made adapters for cameras, microphones, Azure Kinect, HoloLens 2, Mask‑RCNN, and other sensors/effectors.
  • Cross‑platform – built on .NET Standard, runs on Windows and Linux (some components are OS‑specific).

Typical use cases

  • Social robots or embodied conversational agents that need to fuse vision, audio and language in real time.
  • Mixed‑reality task‑assistants (e.g., the SIGMA prototype for MR guidance).
  • Ambient‑intelligence / smart‑space applications that process continuous sensor streams.
  • Research prototypes that combine multiple AI models (e.g., object detection + speech) under strict latency.

Getting started

  1. Install the Microsoft.Psi.* NuGet packages or clone the repo and build the code.
  2. Follow the Brief Introduction tutorial on the wiki to learn the stream model and basic pipeline construction.
  3. Watch the 30‑minute video webinar (or the shorter Tech Minutes) for a visual overview and a hands‑on coding demo.
  4. Explore the sample applications (HelloWorld, SimpleVoiceActivityDetector, WebcamWithAudio, WhatIsThat, HoloLensSample) which are ordered by complexity and include step‑by‑step walkthroughs.
  5. Use PsiStudio for visual debugging, annotation and replay of recorded streams.

Documentation & community

License – Core framework and NuGet packages are under the MIT License. Applications under the Applications/ folder (e.g., SIGMA) use a Microsoft Research research‑only license.

Why it matters – \psi abstracts away the low‑level plumbing of synchronized multimodal streams, letting researchers and developers focus on the AI logic that powers embodied, mixed‑reality, or ambient‑intelligence systems. It is already used by labs at Microsoft Research, Georgia Tech, MIT, CMU, Cornell, and many others.

Related

  • Project
    e-sensing/sits

    SITS is an open‑source R package (with a Python wrapper) for building, regularising, and classifying satellite‑image time‑series data cubes. It connects to many public EO collections, offers a suite of ML/DL models (including GPU‑accelerated CNNs and attention encoders), and provides end‑to‑end tools for training, classification, smoothing, uncertainty estimation, and active learning.

  • Project
    labring/FastGPT

    FastGPT is an open‑source platform for building AI agents and LLM‑driven applications. It provides a visual Flow editor, knowledge‑base (RAG) support, plugin hot‑reloading, debugging tools, and can be run locally via Docker or deployed on Sealos/Kubernetes. A hosted SaaS version and a commercial edition are also offered.

  • Project
    AISBench/benchmark

    A model evaluation tool based on OpenCompass that provides accuracy and performance benchmarking for both local and service-based AI models across text and multimodal tasks.

  • Project
    moeru-ai/airi

    An open-source framework for creating interactive AI virtual characters and VTubers capable of chatting, playing games, and interacting across multiple platforms.

  • Project
    apocas/restai

    RESTai is a self‑hosted AI‑as‑a‑Service platform that lets you create multi‑tenant AI projects (LLM chat, RAG, agents, image generation, etc.) and expose them via a simple REST/OpenAI‑compatible API and a built‑in React dashboard. Install via PyPI or Docker, configure projects, and use features like knowledge‑graph RAG, headless‑browser agents, visual Block logic, prompt versioning, and team‑level branding.