syne-tune/syne-tune

What it solves

Syne Tune addresses the challenge of large-scale hyperparameter optimization (HPO) for machine learning models. It provides a framework to efficiently find the best settings (hyperparameters) for a model to maximize performance, supporting distributed execution across local machines or SLURM clusters to handle computationally expensive tuning jobs.

How it works

The library operates by defining a search space for hyperparameters and using a scheduler to suggest configurations to evaluate. It supports two primary interaction patterns:

• Launch-based tuning: The library manages the execution of training scripts as separate processes, reporting metrics back to the tuner via a Reporter object.
• Ask/Tell interface: Users can directly communicate with the scheduler to get a configuration (ask) and provide the result (tell), which is useful for environments like Jupyter notebooks or custom training loops.

It implements state-of-the-art HPO methods including multi-fidelity optimization, multi-objective optimization, transfer learning, and population-based training.

Who it’s for

Machine learning engineers and researchers who need to perform large-scale, reproducible hyperparameter tuning for complex models, including those using Hugging Face transformers or integrating with tools like Optuna.

Highlights

• Distributed Execution: Ability to run experiments locally or on SLURM clusters.
• Diverse HPO Methods: Support for multi-fidelity, multi-objective, and population-based training.
• Simulation Tools: Includes a collection of blackboxes (surrogate and tabular benchmarks) for efficient HPO simulation.
• Flexible Integration: Offers an Ask/Tell interface and a sampler for Optuna via OptunaHub.