OpenAI Introduces IndQA Benchmark for Indian Languages and Culture

OpenAI has introduced IndQA, a benchmark designed to evaluate how AI models reason about culturally nuanced questions in Indian languages. This tool addresses the saturation of existing multilingual benchmarks and the lack of evaluations that capture the specific cultural, historical, and social contexts of the Indian population.

IndQA Framework and Scope

IndQA evaluates knowledge and reasoning across 10 cultural domains using 2,278 questions written natively in 12 languages. Unlike multiple-choice benchmarks, IndQA focuses on reasoning-heavy tasks that require a deep understanding of local context.

Supported Languages and Domains

The benchmark includes the following languages, specifically including Hinglish to account for common code-switching patterns:

Bengali, English, Hindi, Hinglish, Kannada, Marathi, Odia, Telugu, Gujarati, Malayalam, Punjabi, and Tamil.

The 10 cultural domains covered are:

Architecture & Design
Arts & Culture
Everyday Life
Food & Cuisine
History
Law & Ethics
Literature & Linguistics
Media & Entertainment
Religion & Spirituality
Sports & Recreation

Data Structure

Each datapoint in the IndQA set consists of four components:

Culturally grounded prompt: The question written in an Indian language.
English translation: Provided for auditability purposes.
Rubric criteria: Specific grading guidelines written by experts.
Ideal answer: The expected response based on expert knowledge.

Technical Methodology and Construction

IndQA was developed through a collaborative process involving 261 domain experts, including journalists, linguists, scholars, and artists. The construction process followed a strict pipeline to ensure the benchmark remained challenging for frontier models.

Expert Authoring and Review

Native-level speakers and subject matter experts drafted difficult, reasoning-focused prompts tied to their specific regions and specialties. These responses underwent peer review and iterative fixes before final sign-off.

Adversarial Filtering

To prevent the benchmark from becoming saturated, OpenAI applied adversarial filtering. Questions were tested against GPT-4o, OpenAI o3, GPT-4.5, and GPT-5. Only questions where a majority of these models failed to produce acceptable answers were retained, ensuring the benchmark provides headroom for future progress.

Rubric-Based Grading

Instead of simple matching, IndQA uses a weighted rubric system. Domain experts define criteria for what an ideal answer should include or avoid, and each criterion is assigned a point value. A model-based grader then calculates the final score based on the sum of satisfied criteria points.

Performance Tracking and Limitations

OpenAI uses IndQA to track the progress of its models over time. While the lab notes that OpenAI models have improved significantly on Indian languages, they acknowledge that substantial room for improvement remains.

Key Caveats

Not a Language Leaderboard: Because questions are not identical across all languages, IndQA cannot be used for direct cross-language comparisons of ability. It is intended to measure improvement over time within a specific model family or configuration.
Adversarial Bias: Because the question selection process specifically filtered out questions that OpenAI models could answer, the benchmark is adversarial against those models. This may disadvantage OpenAI models when compared to non-OpenAI models.

Strategic Implications

IndQA serves as a blueprint for creating similar benchmarks in other under-represented languages and cultural domains. By moving beyond translation-based tasks and multiple-choice questions, IndQA aims to provide a "north star" for AI research labs to identify and resolve specific failures in cultural reasoning and linguistic nuance.