FrontierCS/Frontier-CS

What it solves

Frontier-CS addresses the saturation of current AI coding benchmarks. While many models now score highly on textbook-style problems, they often struggle with real-world engineering and research challenges. This project provides a set of unsolved, open-ended, and diverse computer science problems that require deep expertise and cannot be solved by simple pattern matching.

How it works

Frontier-CS acts as a rigorous evaluation framework for AI. It provides three main tracks of problems:

• Research Problems: High-level challenges in systems and ML (e.g., FlashAttention).
• Algorithmic Problems: Complex algorithmic tasks with verifiable continuous scoring.
• Frontier-CS 2.0: Agent-native tasks designed specifically for iterative interaction via the Harbor agent evaluation framework.

The system supports local evaluation using Docker, cloud-based evaluation via SkyPilot, and a Python API for integration into other workflows. It also allows for "unbounded" scoring, which is useful for algorithm evolution frameworks.

Who it’s for

• AI Researchers: Those developing LLMs or agents capable of complex reasoning and coding.
• MLOps Engineers: People testing the limits of frontier models on specialized CS domains.
• Agent Developers: Those building autonomous agents that need to be tested on verifiable, high-difficulty tasks.

Highlights

• Unsolved Challenges: Focuses on problems where no solution has yet achieved a perfect score.
• Agent-Native Evaluation: Integration with Harbor allows agents to receive iterative feedback through submit.sh during trials.
• Diverse Domains: Covers systems, machine learning, algorithms, and security.
• Verifiable Scoring: Uses continuous scoring rather than binary pass/fail, allowing for incremental improvement tracking.