AkshitIreddy/Interactive-LLM-Powered-NPCs

What it solves

This project enables players to have real-time, voice-driven conversations with non-player characters (NPCs) in existing open-world games (such as Cyberpunk 2077, GTA 5, and Assassin's Creed) without needing to modify the game's source code. It replaces the static nature of pre-scripted NPC dialogue with dynamic, LLM-generated responses that are synchronized with facial animations.

How it works

The system operates as an overlay that interacts with the game screen and the player's hardware:

1. Input Processing: Captures player voice via microphone (Speech-to-Text) and player emotions via webcam.
2. Character Identification: Uses facial recognition to identify the NPC on screen. If it is a known side character, it loads specific traits; if it is a background NPC, it generates a unique personality.
3. Dialogue Generation: A Large Language Model (LLM) generates a response based on the character's personality, world knowledge stored in vector databases (RAG), and a pre_conversation.json file containing iconic lines to maintain style.
4. Output Generation: The response is converted to speech (TTS) and used to drive a facial animation video (via SadTalker) based on the NPC's face image.
5. Integration: The system replaces the facial pixels of the NPC in the game view with the generated animation, creating the illusion of the NPC speaking naturally.

Who it’s for

Gamers and modders who want more immersive, unscripted interactions with NPCs in previously released open-world titles.

Highlights

• No Source Code Modification: Works by replacing pixels on the screen rather than modding the game engine.
• Multimodal Interaction: Combines voice input, facial recognition, and webcam-based emotion detection.
• Dynamic Memory: Uses vector stores to provide NPCs with limitless memory of the game world and character-specific knowledge.
• Versatile Compatibility: Designed to work with any game by allowing users to define world and character data files.