Leanstral 1.5 Release: State‑of‑the‑art Formal Verification Model

TL;DR

Leanstral 1.5 is a free, Apache‑2.0‑licensed formal‑reasoning model (119 B total, 6 B active parameters) that reaches 100 % on miniF2F, solves 587/672 PutnamBench problems, sets new state‑of‑the‑art scores of 87 % on FATE‑H and 34 % on FATE‑X, and uncovers real bugs in open‑source repositories.

Model Overview

Leanstral 1.5 builds on the original Leanstral architecture with a three‑stage training pipeline: mid‑training, supervised fine‑tuning, and reinforcement learning with the CISPO algorithm. The model is optimized for proof engineering in Lean 4 and operates with only 6 B active parameters despite a total parameter count of 119 B.

Training Environments

Multiturn environment – The model receives a theorem statement, attempts a proof, receives Lean compiler feedback, and iteratively refines the proof until it compiles or the token budget is exhausted.
Code‑agent environment – Leanstral behaves like a developer in a raw filesystem: it edits files, runs bash commands, and queries the Lean language server for goals, errors, and type information. This enables long‑horizon tasks such as completing partial proofs, generating auxiliary lemmas, and persisting across multiple context‑compaction rounds. The final proofs are verified with Mistral’s fork of SafeVerify.

Benchmark Performance

Leanstral 1.5 was evaluated on four major formal‑reasoning benchmarks.

miniF2F

Result: 100 % saturation on both validation and test sets.
Significance: Demonstrates complete coverage of elementary to IMO‑level problems across algebra, combinatorics, and number theory.

PutnamBench

Result: Solved 587 of 672 problems (≈87 %).
Comparison: Beats Seed‑Prover 1.5 (high setting) by 7 problems while costing roughly $4 per problem versus $300+ for Seed‑Prover’s high‑budget configuration.
Scaling: Pass@8 rises monotonically with token budget: 44 problems at 50 k tokens, 244 at 200 k, 493 at 1 M, and 587 at 4 M tokens.

FATE‑H and FATE‑X

Result: New state‑of‑the‑art scores of 87 % on FATE‑H (graduate‑level abstract algebra) and 34 % on FATE‑X (PhD‑level).
Baseline: Outperforms Goedel‑Architect, Seed‑Prover 1.5, and AxProverBase under comparable conditions (no natural‑language guidance).

FLTEval

Result: Pass@1 improves from 21.9 % to 28.9 %; Pass@8 improves from 31.9 % to 43.2 %.
Cost efficiency: Beats Opus 4.6’s 39.6 % Pass@8 while using roughly one‑seventh the compute cost.

Test‑Time Scaling Behaviour

Leanstral exhibits the strongest test‑time scaling observed in formal‑reasoning models. Increasing the token budget per attempt directly translates into more solved problems, as illustrated by the PutnamBench scaling curve. The model can sustain reasoning over millions of tokens, exemplified by the AVL‑tree proof that required 2.7 M tokens and 22 context compactions.

Code Verification Case Studies

Although trained primarily on mathematics, Leanstral 1.5 shows robust capabilities in software verification.

AVL‑Tree Time‑Complexity Proof

Leanstral proved O(log n) insertion and deletion bounds for a real AVL‑tree implementation.
The proof required structural induction, monadic time tracking, and exhaustive case analysis.
Over 2.7 M tokens and 22 compactions, the model derived a bound of 48 steps per tree height unit plus a constant, then linked height to size via a logarithmic relationship.

Automated Bug Discovery

A pipeline translates Rust code to Lean, generates correctness properties, and attempts up to four proofs per property.
Failure to prove a property triggers four attempts to prove its negation.
Across 57 repositories, 47 properties were violated; 11 corresponded to genuine bugs, 5 of which were previously unreported.
Example: In datrs/varinteger, the sign function overflowed on Std.U64.MAX, causing crashes in debug mode and silent corruption in release mode—an edge case missed by conventional testing.

Getting Started

Leanstral 1.5 is released under the Apache‑2.0 license.

Weights: Available on HuggingFace at mistralai/Leanstral-1.5-119B-A6B.
API: Free endpoint leanstral-1-5 documented at Mistral AI’s model cards.
Recommended client: Mistral Vibe.

Quick Installation Steps

# Install Mistral Vibe
uv tool install mistral-vibe
uv tool update mistral-vibe vibe --setup

# Install Leanstral 1.5 (placeholder command)
/leanstallexit

# Launch the agent
vibe --agent lean

Optional: Install the Lean LSP MCP server for richer language‑server interactions.

[[mcp_servers]]
name = "lean-lsp"
transport = "stdio"
command = "uvx"
args = ["lean-lsp-mcp"]
tool_timeout_sec = 600

After setup, users can ask Leanstral to prove theorems, debug existing proofs, or contribute verified code to repositories.

Implications

Leanstral 1.5 demonstrates that high‑performance formal verification can be achieved with a relatively small active parameter footprint and at open‑source cost levels. Its ability to scale with token budget, solve large‑scale mathematical benchmarks, and discover real software bugs suggests a shift toward practical, widely accessible proof engineering tools for both academia and industry.