lightly-ai/lightly

A python library for self-supervised learning and latent world models on images.

What it solves

LightlySSL simplifies the implementation of self-supervised learning (SSL) for computer vision. It provides a modular, ready-to-use framework that allows developers to train models on raw data without needing manual labels, and even enables the training of "world models" that learn how scenes respond to actions.

How it works

The framework acts as a collection of building blocks for PyTorch. It exposes low-level components like loss functions and model heads, allowing users to plug in custom backbone models. It supports distributed training through PyTorch Lightning and provides pre-built implementations of many state-of-the-art SSL algorithms, such as DINOv2, MoCo, and SimCLR. For world models, it uses techniques like LeWM to learn from raw frames and actions, enabling agents to plan by imagining outcomes in an embedding space.

Who it’s for

This project is designed for computer vision researchers and machine learning engineers who want to implement self-supervised learning or world models in their PyTorch projects without building complex training loops from scratch.

Highlights

  • Modular design with low-level building blocks for custom model development.
  • Extensive library of pre-implemented state-of-the-art models (e.g., DINOv2, MAE, SimCLR).
  • Native support for distributed training via PyTorch Lightning.
  • Includes tools for training world models that can plan actions based on visual goals.
  • Provides numerous tutorials and Google Colab notebooks for quick experimentation.

Related

  • Project
    vturrisi/solo-learn

    A PyTorch Lightning-powered library of self-supervised methods for unsupervised visual representation learning, providing a unified environment to train and compare SOTA visual models.

  • Project
    galilai-group/stable-pretraining

    stable‑pretraining is a PyTorch‑Lightning (and experimental JAX) library that lets you plug‑in self‑supervised learning methods by defining only a forward function. It handles data loading (with optional GPU‑side augmentations), checkpointing, multi‑GPU training, and a rich set of online evaluation callbacks, making SSL research faster and more reproducible.

  • Project
    Lightning-AI/pytorch-lightning

    A deep learning framework that abstracts away the engineering boilerplate of PyTorch, enabling researchers to pretrain and finetune AI models at scale across various hardware.

  • Project
    lightly-ai/lightly-train

    LightlyTrain is a framework for the entire computer vision model lifecycle, enabling users to pretrain, fine-tune, and distill state-of-the-art models for detection and segmentation tasks with support for edge deployment.

  • Project
    kengz/SLM-Lab

    A modular PyTorch framework for Deep Reinforcement Learning that allows users to configure and run RL experiments via JSON specs across various Gymnasium environments.