prov-gigatime/GigaTIME

GigaTIME: Multimodal AI generates virtual population for tumor microenvironment modeling (Cell)

What it solves

GigaTIME is designed to generate "virtual population" profiles for tumor microenvironment modeling. Specifically, it solves the problem of predicting spatial proteomics (mIF) from routine, widely available H&E pathology slides, removing the need for expensive and complex spatial proteomics experiments.

How it works

The project provides a cross-modal translator that transforms H&E stained pathology images into virtual multiplexed immunofluorescence (mIF) maps. It includes two versions of the model:

  • GigaTIME: The original CNN-based model.
  • GigaTIME-Flash: An efficient version built on GigaPath-Flash that offers 6× faster inference, 8× less GPU memory usage, and improved prediction quality.

Who it’s for

AI researchers specializing in pathology and medical imaging who want to reproduce the experimental results from the associated Cell paper or build upon this work for tumor microenvironment analysis.

Highlights

  • Cross-modal translation: Converts routine H&E slides to virtual mIF profiles.
  • Efficient inference: GigaTIME-Flash significantly reduces computational overhead while improving quality.
  • Whole-slide capability: Supports tiling and stitching for slide-level virtual mIF map generation.
  • Research-focused: Specifically designed for tumor microenvironment modeling in research settings.

Related

  • Project
    prov-gigapath/prov-gigapath

    A whole-slide foundation model for digital pathology that uses a tile-and-slide encoder architecture to analyze massive pathology images.

  • Project
    rapidsai/cucim

    cuCIM is a GPU-accelerated computer vision and image processing library designed for large, multidimensional images used in biomedical, geospatial, and remote sensing applications.

  • Project
    HUANGLIZI/LViT

    LViT is a multimodal framework that combines language and vision transformers to improve the accuracy of medical image segmentation in CT scans and other medical imagery.

  • Project
    rendeirolab/LazySlide

    A Python framework for whole slide image analysis that integrates with the scverse ecosystem to enable interoperable and scalable histological workflows.

  • Project
    jd-opensource/JoyAI-Image

    A unified multimodal foundation model that integrates image understanding, text-to-image generation, and instruction-guided image editing with a focus on spatial intelligence.