Simreal-AI/Simreal-MLBench

What is this?

Simreal-MLBench is a benchmark for evaluating AI research agents — AI systems that can do real machine-learning research work, not just answer questions. An agent is given a dataset, must figure out a modeling approach, run experiments, and submit predictions, just like a human data scientist would.

The project is a public preview of a commercial evaluation service. The key idea: the rules are open for anyone to inspect, but the infrastructure that enforces them is run by the company (Simreal).

What does it contain?

• 60 tasks spanning tabular learning, forecasting, vision, language, audio, multimodal learning, and scientific data. Tasks are split into three tiers: Easy (20 tasks, 6-hour budget, CPU-only), Medium (20 tasks, 12 hours, GPU), and Hard (20 tasks, 24 hours, GPU).
• Public evaluation protocol — the exact scoring code, task definitions, and a client SDK (mleb-remote) that agents use to interact with the evaluation environment.
• Verification tooling — scripts that let anyone recompute published scores from saved data, so results don't require trusting the company.

Why is it designed this way?

The authors argue benchmark numbers are only trustworthy if you can verify them. Their design choices:

1. The scorer isn't them. Test labels stay with the original competition platform (e.g., Kaggle). Simreal never holds the answers, so neither agents nor the company can leak or mis-split them.
2. Two submissions, not one. Agents get a first score, then a second chance. This measures not just raw skill but whether an agent can improve after seeing honest feedback.
3. Source-blind task briefs. Briefs are scanned to remove platform names and URLs, so agents can't easily recognize a well-known dataset.
4. Adversarial testing. Before a task ships, the scoring rewards are probed with deliberately bad submissions to catch exploits.
5. Full provenance. Every score carries a submission ID, artifact hash, and snapshot hash — a complete audit trail.

Scoring

A task score is 100 × p², where p is the fraction of a frozen reference leaderboard that the agent outperformed (ties get half credit). This rewards beating many baselines, not just topping one.

What's open vs. closed?

Open (Apache-2.0)	Closed (Simreal service)
Task catalog, protocol, scoring code, SDK, examples, tests, report	Gateway, controller, worker queues, submission broker, official scoring route

Status

The catalog and tooling are released (35 tests passing), but only 3 of 60 tasks are verified end-to-end so far. Full-suite admission and a public leaderboard are planned. The README also mentions future research directions: using the benchmark for RLVR (reinforcement learning with verifiable rewards) and recursive self-improvement — agents improving their own research skills over time — though no trained results are included yet.

Bottom line

A serious, well-thought-out evaluation framework for ML research agents, inspired by OpenAI's MLE-bench but with a distinct scoring scheme, resource protocol, and a deliberate open-rules/operated-runtime split. Still early-stage: the full benchmark isn't verified end-to-end yet, and using the official evaluation service requires contacting the company.