optuna/optuna

A hyperparameter optimization framework

What it solves

Optuna is an automatic hyperparameter optimization (HPO) framework designed specifically for machine learning. It automates the process of finding the best set of hyperparameters for a model, removing the manual trial-and-error process of tuning parameters to improve model performance.

How it works

Optuna uses a "define-by-run" API, which allows users to dynamically construct search spaces using standard Python syntax (such as loops and conditionals). The framework organizes optimization into "studies" (the overall optimization task) and "trials" (individual executions of the objective function). It employs state-of-the-art sampling algorithms to suggest new hyperparameters and pruning techniques to stop unpromising trials early to save compute resources.

Who it’s for

It is intended for machine learning practitioners and researchers who need to efficiently tune hyperparameters for their models across various platforms and scales.

Highlights

  • Pythonic Search Spaces: Define hyperparameters using familiar Python code, allowing for complex, conditional search spaces.
  • Efficient Optimization: Uses advanced sampling and pruning algorithms to ensure faster convergence to optimal parameters.
  • Easy Parallelization: Scale optimization studies across tens or hundreds of workers with minimal code changes.
  • Integrated Visualization: Includes plotting functions and a dedicated real-time web dashboard for inspecting optimization history and parameter importance.
  • Rust-based Implementation: Offers an experimental Rust-based version (Rustuna) for increased sampling speed and memory efficiency.

Related

  • Project
    PrunaAI/pruna

    A model optimization framework that uses compression algorithms like quantization and pruning to make AI models faster, smaller, and cheaper to run.

  • Project
    syne-tune/syne-tune

    A library for large-scale and reproducible hyperparameter optimization (HPO) that supports distributed execution and state-of-the-art tuning methods.

  • Project
    mljar/mljar-supervised

    An Automated Machine Learning package for tabular data that automates preprocessing, model selection, and tuning while providing transparent, detailed reports for explainability.

  • Project
    hyperactive-project/Hyperactive

    A unified Python interface for hyperparameter tuning and black-box optimization that separates optimization problems from algorithms across multiple backends.

  • Project
    triton-inference-server/model_analyzer

    A CLI tool for Triton Inference Server that automates the search for optimal model configurations to balance performance and resource usage on specific hardware.