FrontierCS/Frontier-CS

A benchmark for evaluating LLMs on open-ended CS problems. Exploring the Next Frontier of Computer Science.

What it solves

Frontier-CS addresses the saturation of current AI coding benchmarks. While models often score highly on textbook-style problems with known solutions, they frequently struggle with real-world engineering and research challenges. This project provides a set of unsolved, open-ended, and diverse computer science problems that require deep expertise and lack optimal solutions, ensuring that AI evaluation remains challenging as models evolve.

How it works

Frontier-CS functions as a rigorous "exam" for AI, offering three primary tracks of problems: research problems, algorithmic problems, and agent-native 2.0 problems. It provides a CLI and Python API to evaluate solutions, supporting both local execution via Docker and cloud-based execution via SkyPilot. To support iterative AI agents, it integrates with Harbor, allowing agents to receive feedback through a submit.sh script and receive a final verifiable score.

Who it’s for

This tool is designed for AI researchers and developers building LLMs or AI agents focused on coding, problem-solving, and computer science research, as well as those looking to benchmark their models against genuinely difficult, non-trivial tasks.

Highlights

  • Unsolved Challenges: Focuses on problems where no solution has yet achieved a perfect score.
  • Verifiable Scoring: Uses continuous scoring rather than binary pass/fail, allowing for incremental improvement tracking.
  • Diverse Scope: Covers multiple domains including systems, machine learning, algorithms, and security.
  • Agent-Native Integration: Specifically designed for agentic workflows via Harbor adapters, supporting iterative feedback and trials.
  • Flexible Execution: Supports batch evaluation and multiple backends (Docker, SkyPilot) for scalable testing.

Related

  • Project
    harbor-framework/terminal-bench-science

    A benchmark for evaluating AI agents on expert-curated research workflows across life, physical, earth, mathematical, and engineering sciences.

  • Project
    harbor-framework/terminal-bench

    Terminal-Bench is a continuous benchmark for measuring the capabilities of frontier AI agents using a diverse and evolving set of difficult tasks.

  • Project
    scaleapi/SWE-bench_Pro-os

    A challenging benchmark for evaluating LLMs and AI agents on long-horizon software engineering tasks by requiring them to generate patches for real-world codebase issues.

  • Project
    UKGovernmentBEIS/inspect_evals

    A library of evaluations built using the Inspect AI framework to measure the capabilities and risks of frontier AI models.

  • Dispatch
    Frontier AI on Your Own Hardware: Open‑Source Week Shows How Small Labs Can Compete

    Tim Dettmers’ Open‑Source Week demonstrates that university labs with a few GPUs can run frontier‑scale models locally, build autonomous research agents, and achieve cost‑effective scaling, challenging the belief that only massive GPU farms can do cutting‑edge AI.