undreamai/LLMUnity

Create characters in Unity with LLMs!

What it solves

LLMUnity allows game developers to integrate Large Language Models (LLMs) directly into the Unity engine. It solves the difficulty of creating intelligent, interactive AI characters that can chat with players in real-time without requiring constant internet access or expensive cloud API calls.

How it works

The project uses a backend called LlamaLib (built on llama.cpp) to run LLMs locally on the user's machine. It provides Unity-specific components like LLM for model management and LLMAgent for character behavior. Developers can load .gguf models, define system prompts to set character roles, and use C# scripts to trigger chats. It also supports a remote server configuration where the LLM runs on a separate machine and the Unity client connects via IP.

Who it’s for

Unity game developers and interactive experience creators who want to add AI-driven NPCs, chatbots, or dynamic storytelling to their projects across PC, mobile, or VR platforms.

Highlights

  • Local Execution: Runs locally on CPU and GPU (Nvidia, AMD, Apple Metal), ensuring data privacy and offline availability.
  • Cross-Platform: Compatible with PC, mobile (iOS/Android), and VR.
  • RAG System: Includes a Retrieval-Augmented Generation system for semantic search to enhance character knowledge.
  • Flexible Control: Supports grammar restrictions (GBNF) for structured output (like JSON) and function calling.
  • Model Support: Compatible with all major LLM models in GGUF format.