Building a Low-Latency AI Gaming Companion for Skyrim

Developer pantelisk has developed Varkos, an intelligent gaming companion designed to play alongside the user in Skyrim. Unlike traditional AI NPC frameworks that focus primarily on dialogue, Varkos is built for high world agency and low latency, allowing the companion to perform complex, multi-step physical actions within the game world based on natural language commands.

Low-Latency Architecture and Performance

To avoid the immersion-breaking delays common in cloud-based LLM implementations, Varkos utilizes a hybrid local-inference stack. The system is designed to start processing the player's speech as soon as it begins, aiming for a total response time of under 500ms in optimal cases.

The Technical Stack

  • Audio Processing: The system uses an optimized Qwen3-ASR 1.7b model with custom kernels. A custom harness processes audio in rolling partials (40-80ms) to support streaming voice-to-text.
  • Voice Activity Detection (VAD): Optimized versions of Turnpipe and Silero are used to determine when a player has finished speaking or is interrupting the AI.
  • Audio Generation: Varkos employs PocketTTS-Raven for fast generation (20-30ms) and Qwen-3-TTS for higher emotional control, depending on the complexity of the response.
  • Hardware: The game runs on Windows, while the "brain" and audio processing run on an M4 MacBook (though the author notes it could run entirely on Windows with ~12GB of dedicated GPU RAM).

Latency Budget Breakdown

Component Latency
Voice to Text 40-80ms
Action Analysis 20ms
Response Generation & Grounding 300-600ms
Audio Generation 20-60ms

World Agency via Action Latent Encoder (ALE)

The core differentiator of Varkos is the Action Latent Encoder (ALE), a hybrid system combining embeddings, small classifiers, explicit rules, and traditional machine learning. ALE allows Varkos to translate natural language into grounded game actions without relying solely on a large LLM for every decision.

Key Capabilities

  • Complex Command Decomposition: Varkos can handle multi-step instructions. For example, a command to "wait here, then bring a potion after an arrow signal" involves registering a future trigger and monitoring the game state for a specific event.
  • Grounded Item Search: Instead of hallucinating items, Varkos searches the actual game world JSON state to identify and retrieve specific objects.
  • Persistent Goals: Activities like "hide-and-seek" are treated as persistent plans with movement and completion conditions rather than single API calls.
  • Phrasing Invariance: ALE is designed to be invariant to phrasing; whether a user says "grab the sword" or "fetch the damn sword," the system maps the request to the same action prototype.

Personality Evolution and "Void Mode"

While real-time actions are handled locally, Varkos uses cloud LLM calls for slow-burn personality evolution. This process happens asynchronously, updating the companion's traits and emotional homeostasis based on shared experiences.

Dynamic Character Growth

Varkos begins as a sarcastic, proud demon reincarnated as a dog. Over time, shared experiences can lead him to become more domesticated and affectionate, changing his vocabulary and behavior based on the player's treatment of him.

Cross-Game Persistence (Void Mode)

Varkos exists as a standalone entity outside of Skyrim. When the game closes, he enters "Void Mode," where he can still communicate via speech. This architecture allows the companion to potentially follow the player across different games, adapting to new environments and rules as they are transitioned.

Community Insights and Technical Discussion

Community members on Hacker News highlighted the emotional resonance of the "lovable sidekick" persona, noting that the AI's occasional failures feel natural to the character's personality.

"What's great is that the LLMisms fold neatly into 'dumb but lovable sidekick.' So even when it chokes or stumbles on a command, the kind of frustration the user expresses when correcting it feels natural and part of the game even."

Other technical discussions focused on the potential for this approach to be integrated into console gaming via dedicated AI hardware or the possibility of replacing ALE with future "Live" multimodal models that can dispatch tools asynchronously while speaking.

Sources

Related

  • Project
    dnhkng/GLaDOS

    A multi-modal, autonomous AI persona framework that creates a proactive voice assistant with vision, long-term memory, and a low-latency response pipeline.

  • Project
    Orkas-AI/Orkas

    Orkas is a cross‑platform desktop app that lets you run a local AI workforce. A central “Commander” agent takes a natural‑language goal, creates a plan, and dispatches specialist agents (coding, research, data analysis, video, etc.) to complete the work. All data, files, and API keys stay on your machine; the app simply forwards model calls to the providers you configure (OpenAI, Claude, Gemini, etc.). Features include per‑agent private memory, visibility slicing to limit token usage, self‑evolution via meta‑files, and integration with external CLI coding agents. Installers are available for macOS and Windows; Linux runs from source. MIT‑licensed.

  • Project
    agiresearch/AIOS

    AIOS is an open‑source “operating system” for LLM‑based agents. The kernel (this repo) manages LLM instances, memory, storage, and tool sandboxing, exposing a syscall‑style API via the Cerebrum SDK. It supports many model back‑ends (Ollama, vLLM, HuggingFace, Deepseek), integrates external tools, and offers several deployment modes—from local single‑machine setups to remote, personal, and virtualized kernels. Installation is via a Python 3.10/3.11 environment, configuration is done in `config.yaml`, and agents can be run through a web UI or a natural‑language terminal. The project is research‑grade (multiple conference papers) and actively maintained.

  • Project
    SlimeBoyOwO/LingChat

    An immersive AI companionship assistant that combines LLMs with emotion recognition, desktop vision, and Galgame-style visuals to create a reactive digital character.

  • Project
    GetStream/Vision-Agents

    A framework for building low-latency, multi-modal AI agents that can watch, listen, and understand video in real time using a combination of CV models and LLMs.