TextQuests: Evaluating LLM Agentic Reasoning in Text-Based Video Games

TextQuests benchmark for autonomous agent evaluation

Hugging Face has introduced TextQuests, a benchmark designed to evaluate the ability of Large Language Models (LLMs) to act as autonomous agents in complex, exploratory environments. While frontier models have saturated static knowledge benchmarks like MMLU and GPQA, TextQuests tests whether this knowledge translates to dynamic, interactive settings that require sustained, self-directed reasoning over growing contexts.

Benchmark design and methodology

TextQuests utilizes 25 classic Infocom interactive fiction games. These games serve as a rigorous testbed because they often require hundreds of precise actions and can take human players over 30 hours to complete. The benchmark specifically measures two core agentic capabilities:

  • Long-Context Reasoning: The ability to devise and execute multi-step plans based on a continuously growing history of actions and observations without external tools.
  • Learning through Exploration: The capacity to learn from experience, analyze failures, and make incremental improvements through trial-and-error.

Evaluation metrics

Models are evaluated across two distinct runs: one with access to official game hints ("With Clues") and one without ("No Clues"). Each run is limited to 500 steps, with the full game history maintained without truncation to test long-context performance. Performance is measured using two primary metrics:

  1. Game Progress: Measured by the number of labeled checkpoints (necessary objectives) the agent completes on the path to finishing the game.
  2. Harm: An ethical assessment that tracks specific in-game actions considered harmful, averaged across all games to determine the agent's overall tendency toward harmful behavior.

Key findings on LLM performance

Long-context reasoning failures

As the context window exceeds 100K tokens, current LLMs exhibit significant failures in precise reasoning and planning. Common issues include:

  • Hallucinations: Models often hallucinate prior interactions, such as believing they have collected an item they actually have not.
  • Repetitive Behavior: Agents show an increased tendency to repeat previous actions rather than synthesizing new plans as the context lengthens.
  • Spatial Reasoning Deficits: Models struggle with mental mapping. For example, in the game Wishbringer, LLMs frequently failed to navigate back down a cliff by reversing their ascent path—information that was explicitly available in the context history. Similarly, all frontier LLMs struggled to navigate the Maze in Zork I.

Dynamic thinking and efficiency

The benchmark reveals a trade-off between reasoning depth and operational efficiency. While models that utilize more test-time compute (generating more reasoning tokens) generally achieve higher performance, this benefit diminishes after a certain budget.

Because many exploratory steps, such as basic navigation, are intermediate and do not require deep reasoning, the ideal LLM agent is one that can dynamically adjust its reasoning effort based on the task complexity to optimize inference cost and latency.

Sources