Mistral AI Rails Testing Agent

Mistral AI has developed an autonomous agent designed to eliminate the testing gap in large Ruby on Rails monoliths by automatically generating and improving RSpec tests. The agent operates within CI/CD pipelines without human intervention, utilizing parallel execution to handle large-scale codebases.

Autonomous Test Generation Workflow

The agent employs a structured execution plan to ensure comprehensive test coverage and adherence to style guidelines. This process is managed via a repository-level AGENTS.md file that directs the agent through the following steps:

1. Source Analysis: Read the source file and any available documentation.
2. Existing Test Check: Determine if a spec already exists.
3. Skill Selection: Select a specialized skill based on the source file location.
4. Pattern Identification: Identify existing patterns, factories, and helpers.
5. Execution: Extract logic, manage factories, and generate tests.
6. Validation: Verify the code using RuboCop and SimpleCov tools.

To prevent the agent from skipping public methods, the workflow includes a mandatory self-review step where the agent must count and verify that every public method in the source file has been tested.

Technical Implementation via Vibe

The agent is built on Vibe, Mistral's open-source coding assistant. The implementation relies on three primary levers: context engineering, specialized skills, and custom tools.

Context Engineering and Specialized Skills

Because different Rails components require different testing strategies, the agent uses separate "skills" files for each category (e.g., models, controllers, serializers, mailers, helpers, and plain Ruby files). For example, a controller skill enforces specific rules such as asserting JSON content rather than just HTTP status codes and verifying authentication for every action.

Custom Validation Tools

To ensure the generated tests are functional and compliant, the agent utilizes custom tools integrated into the Vibe framework:

• RuboCop Tool: A linting tool that detects style violations, triggering the agent to rewrite the code until it is compliant.
• SimpleCov Tool: This tool executes the spec file via RSpec and reports both the pass/fail results and the line coverage percentage. This step is critical because it prevents the "missing parenthesis problem," where a test appears qualitatively high-quality to an LLM but fails to execute due to syntax errors.

Measuring Test Quality

Mistral AI uses a hybrid approach to evaluate the quality of the generated tests, combining quantitative tool-based metrics with qualitative LLM-based scoring.

Quantitative Metrics

The agent tracks the following metrics using standard Ruby tools:

• RSpec: Pass/fail status of tests.
• RuboCop: Number of style violations per file.
• SimpleCov: Percentage of code coverage.

LLM-as-a-Judge

To assess qualitative aspects—such as whether error conditions are tested or if assertions are precise—Mistral uses an "LLM-as-a-judge" model. This judge uses a strict scoring rubric (0.0 to 1.0) based on the presence of precise values (e.g., eq(100) instead of be_present) and the coverage of happy, error, and boundary paths.

Experimental Results

In an experiment involving a repository with 275 source files, the agent was tasked with generating tests for uncovered files and improving existing ones. The results demonstrated a significant increase in test quality and coverage:

Metric	Result
Files processed	275
Tests passing	100%
Average line coverage (SimpleCov)	100%
RuboCop violations after self-correction	0
LLM-as-a-judge score	0.74

Performance by File Type

The agent's effectiveness varied by the type of logic being tested, with self-contained business logic being the easiest to automate:

• Models: 0.81 score
• Serializers: 0.80 score
• Controllers: 0.67 score

Mistral AI noted that only one-third of tests passed on the first execution, highlighting that the iterative self-correction loop powered by SimpleCov and RSpec was the most impactful part of the system design.