official-stockfish/fishtest

What it solves

Fishtest provides a distributed system for testing chess engines, allowing developers to crowdsource the massive amount of computing power needed to verify if a new code change (patch) actually improves the engine's playing strength.

How it works

The system consists of a server and a distributed network of workers:

• Workers: These are installed on contributor computers. They automatically download the engine sources, the current Stockfish version, and the the patch being tested. They compile the code for their specific platform, run batches of games using the fastchess manager, and upload the results back to the server.
• Server: The server manages the test queue and priorities. It calculates probabilistic values from the game results and monitors tests until they reach statistical significance, at which point it publishes the final results.

Who it’s for

Chess engine developers and CPU contributors who want to collaborate on the improving the strength of Stockfish.

Highlights

• Distributed Task Queue: Leverages a crowd of contributors to perform large-scale testing.
• Automated Pipeline: Handles everything from source downloading and compilation to game execution and results reporting.
• Crosspatform Support: Workers can be compiled for different platforms.
• Statistical Validation: Automatically stops tests once results are statistically significant.