alexziskind1/codeneedle

What it solves

CodeNeedle is a benchmark designed to measure an LLM's "positional recall" in long-context windows. Unlike simple named-entity lookup, it tests whether a model can accurately reproduce the first N lines of a specific function's body verbatim from a large source code corpus stuffed into its context.

How it works

The tool loads a source code corpus (Python or JavaScript) into the LLM's context and prompts the model to reproduce a specific function's body based on its verbatim signature. It then compares the model's output against the ground truth source code using a scoring policy that ignores blank lines (to prevent inflating scores) but can optionally include comments and docstrings. The project provides both real-world corpora (like jQuery and Python's http_server) and "novel" synthetically generated corpora to prevent models from relying on training data memory.

Who it’s for

• AI Researchers and Developers testing the effective context window and retrieval capabilities of LLMs.
• Model Evaluators comparing different model architectures, quantizations, or reasoning capabilities (e.g., comparing reasoning vs. non-reasoning models).

Highlights

• Novel Corpora: Includes deterministically generated Python code with opaque identifiers to ensure models aren't just recalling training data.
• Crashes Safety: Results are written atomically after every query, allowing runs to be resumed without losing hours of inference.
• Strict Scoring: Implements a refined scoring system that excludes blank lines and handles indentation variations to ensure accuracy.
• Visualization: Includes a Plotly-based dashboard builder to visualize and compare performance across different models and corpora.
• Multi-Language Support: Supports .py, .js, .mjs, and .cjs files.