Hugging Face NPC-Playground: Integrating LLM-Powered NPCs in 3D Environments

Hugging Face has introduced NPC-Playground, a 3D interactive demo that allows users to engage with Non-Playable Characters (NPCs) powered by Large Language Models (LLMs). This project demonstrates how LLMs can replace traditional rule-based systems to create characters capable of realistic conversation, complex action execution, and instruction following.

The NPC-Playground Technical Stack

NPC-Playground is built using three primary components to bridge the gap between language models and 3D game environments:

  • Cubzh: A cross-platform, open-source User Generated Content (UGC) game engine designed as an alternative to Roblox. It provides the 3D environment, voxel items, and a Lua scripting API for game logic.
  • Gigax: An engine specifically designed for running LLM-powered NPCs at scale. It handles the intelligence and action-selection logic for the characters.
  • Hugging Face Spaces: The hosting environment used to deploy and iterate on the game concepts in an open-source manner.

Gigax: Powering NPC Intelligence via Function Calling

Gigax enables NPCs to interact with their environment by utilizing a "function calling" principle. This approach transforms the LLM from a simple text generator into an agent capable of executing specific in-game actions.

Input and Output Flow

The system processes NPC behavior through a structured pipeline:

  1. Input: The model receives a text description (prompt) of the 3D scene, a summary of recent events, and a comprehensive list of available actions (such as <say>, <jump>, or <attack>).
  2. Output: The model generates a specific action command using parameters that refer to existing 3D entities in the scene. An example output would be say NPC1 "Hello, Captain!".

Open-Source Models

Gigax has open-sourced its inference stack and provided fine-tuned models on the Hugging Face Hub for developer use:

  • NPC-LLM-3_8B: A fine-tuned version of the Phi-3 model.
  • NPC-LLM-7B: A fine-tuned version of the Mistral-7B model.

Cubzh: The UGC Game Engine

Cubzh serves as the underlying game engine for the demo. It is a public Alpha project that allows users to create world items and avatars using a library of over 25,000 community-made voxel items. Developers can code game experiences using a Lua scripting API, which in the NPC-Playground demo allows users to teach NPCs new skills by modifying the cubzh.lua file.

Customization and Developer Resources

Users can interact with the live demo via Hugging Face Spaces. For those looking to build their own AI-powered NPCs, Hugging Face provides a comprehensive "ML for Games Course" tutorial and detailed documentation on tweaking NPC behavior and adding new skills.

Sources