benchflow-ai/awesome-evals

What it is

benchflow-ai/awesome-evals is a curated list of resources about building and evaluating AI agents. It collects papers, blog posts, talks, courses, tools, and benchmarks, and adds a short note to every entry explaining what it is and why it matters. The project is maintained by BenchFlow, a company, and welcomes contributions from others.

What makes it different from a plain link list

The README emphasizes that this is not a simple link dump. Every resource gets a description, every URL is checked, quotes from the sources are kept verbatim, and dead or abandoned tools are removed rather than silently left in place. There are currently 443+ curated links and 143 deep reading notes. Two markers are used: 🆕 means the resource was released or updated in 2025–2026, and ⚠️ flags a caveat (for example, an unverified URL).

How the list was built

According to the README, the curators assembled it using several methods:

• A depth-4 recursive citation crawl that examined 11,600 papers and ranked them by how often they were cited, to surface the academic core of the field.
• Targeted discovery of industry sources that citation graphs miss — blogs from well-known practitioners.
• 47 talks and podcasts that were transcribed and summarized in detail, with verbatim quotes and timestamps.
• Per-section gap audits with adversarial verification, meaning each section was checked for missing material.

What's inside

The list is organized into sections, including:

• Why we need evals (the motivation for evaluating AI agents)
• Why being able to evaluate a capability is the same as having built it (with connections to reinforcement learning)
• A model / harness / skill decomposition
• Observability and what surfaces of an agent you can grade
• Evaluation infrastructure (datasets, scorers, tracing, CI)
• Benchmarks vs. evals, and benchmark integrity issues like contamination and leaderboard gaming
• Evals and reinforcement learning environments
• LLM-as-judge and verifiers
• Agent-specific evaluation (trajectories, tool use, multi-turn, world state)
• Safety and adversarial evaluation (prompt injection, jailbreaks)
• Talks, podcasts, and slides
• Companies and landscape overviews

Also included: a playbook

Beyond the list itself, the repo contains PATTERNS.md, a playbook with real, runnable code and worked examples for techniques like LLM-as-judge (aligned to human judgments), pass@k/pass^k metrics, error analysis, trajectory and world-state grading, CI gating, and verifiable rewards.

A starting set

The README opens with a "must-read starter set" of 12 resources. These include:

• Shunyu Yao's essay "The Second Half" (evaluation becomes more important than training)
• Eugene Yan on fixing your evaluation process rather than buying tools
• Han-Chung Lee on hidden technical debt in agent evaluation infrastructure
• Hamel Husain & Shreya Shankar's LLM Evals FAQ
• Jason Wei on the asymmetry of verification
• Anthropic's guide to evals for AI agents
• Ofir Press on building good language modeling benchmarks
• A paper from Princeton researchers on cost as a first-class metric
• Nathan Lambert on evaluation quicksand
• An academic paper on "Who Validates the Validators?"
• Florian Brand's "Benches 2026" on benchmarks in the agent era
• OpenAI on trustworthy third-party evaluations

Each of these is annotated in the list with a one-line summary of its key point.

Practical details

The project uses an Awesome-style format (the awesome.re badge). It has a contributing guide, a license, and a Discord community. The README suggests reading the must-read starter set first if you're new to the topic.