Qwen-Agent: Generalizing LLMs from 8k to 1M Context

Qwen has introduced a method to enable Large Language Models (LLMs) with a limited 8k context window to understand and process documents containing up to 1 million tokens. By building a multi-level agent architecture, Qwen can synthesize high-quality long-context training data to eventually fine-tune a native 1M-context chat model.

The Three-Level Agent Architecture

To overcome the limitations of short context windows, Qwen developed an agent system that increases in complexity across three levels, each addressing a specific failure mode of long-context processing.

Level 1: Keyword-Based Retrieval-Augmented Generation (RAG)

Level 1 employs a traditional RAG approach to fit relevant information into an 8k-token window. To improve retrieval accuracy, the agent follows a three-step process:

  1. Query Decomposition: The model separates user instructions (e.g., "reply in English") from the actual informational query (e.g., "when were bicycles invented").
  2. Keyword Extraction: The model deduces multilingual keywords from the informational query.
  3. BM25 Retrieval: The agent uses the BM25 algorithm to locate the most relevant chunks of text based on those keywords.

Qwen noted that vector-based retrieval did not provide significant enough improvements to justify the added complexity of deploying a separate embedding model.

Level 2: Chunk-by-Chunk Reading

To prevent the "missing needle" problem where relevant chunks lack keyword overlap with the query, Level 2 uses a brute-force parallel processing strategy:

  1. Relevance Assessment: Every 512-token chunk is processed in parallel. The model assesses each chunk and either outputs "None" or extracts the relevant sentences.
  2. Refined Retrieval: The extracted relevant sentences are used as a new search query to retrieve the most relevant chunks via BM25.
  3. Final Generation: The final answer is generated based on these refined retrieved chunks.

Level 3: Step-by-Step Reasoning

To handle multi-hop reasoning—where an answer requires connecting multiple disparate facts—Level 3 wraps the Level 2 agent as a tool within a ReAct-style tool-calling agent.

When faced with a complex question, the Level 3 agent decomposes the problem into sub-questions. It iteratively queries the Level 2 agent, adding each response to its memory until it has sufficient information to provide a final answer. For example, to find a vehicle invented in the same century as Beethoven's Fifth Symphony, the agent first identifies the century of the symphony and then searches for vehicles invented in that specific century.

Performance Benchmarks

Qwen tested the agent using a 7B chat model (the "32k-Model") and compared it against a standard RAG implementation ("4k-RAG") and the full agent strategy ("4k-Agent"). Tests were conducted on NeedleBench and LV-Eval benchmarks for 256k-context.

Key Findings:

  • Short Contexts: The 32k-Model may outperform 4k-RAG due to the model's better ability to handle multiple parts of a short context.
  • Long Contexts: As document length increases, 4k-RAG typically outperforms the 32k-Model, indicating that the 32k-Model was not optimally trained for long contexts.
  • Agent Superiority: The 4k-Agent consistently outperformed both the 32k-Model and the 4k-RAG across the benchmarks.

Additionally, the agent successfully passed a "needle in a haystack" pressure test with 1 million tokens, though Qwen noted the current lack of reliable quantitative benchmarks for 1M-token real-world applications.

Implications for Model Training

This agent framework serves as a data generation engine. By recording interactions between volunteers and the agent or using the agent to cross-validate other synthetic data, Qwen can create high-quality, long-context fine-tuning datasets. This process allows for the distillation of agent-level capabilities into a native 1M-context chat model, effectively using a "weak" 8k-context model to train a "strong" long-context model.

Sources