HUANGLIZI/LViT

[IEEE Transactions on Medical Imaging/TMI 2023] This repo is the official implementation of "LViT: Language meets Vision Transformer in Medical Image Segmentation"

What it solves

LViT addresses the challenge of medical image segmentation by combining language and vision transformers. It aims to improve the accuracy of delineating anatomical structures or lesions in medical scans (such as CT scans) by leveraging textual descriptions alongside visual data.

How it works

The project implements a Vision Transformer (ViT) based architecture that integrates language information to guide the segmentation process. It supports pre-training and fine-tuning workflows, allowing the model to be initialized with pre-trained weights to achieve higher performance on specific medical datasets.

Who it’s for

This tool is designed for medical imaging researchers and AI developers working on automated medical image segmentation, particularly those focusing on COVID-19 CT scans or esophageal cancer radiotherapy.

Highlights

  • Multimodal Integration: Combines language and vision transformers for improved segmentation.
  • Detailed Benchmarking: Provides performance metrics (Dice and IoU scores) across multiple datasets including QaTa-COV19, MosMedData+, and MoNuSeg.
  • Pre-training Support: Includes options for pre-training the model to enhance final segmentation results.
  • Reproducibility Focus: Implements deterministic mode and random seed settings to ensure consistent results across experiments.

Related

  • Project
    lucidrains/vit-pytorch

    A comprehensive PyTorch library implementing the original Vision Transformer (ViT) and dozens of its advanced variants for image classification.

  • Dispatch
    Fine-Tuning Vision Transformer (ViT) for Image Classification with Hugging Face Transformers

    Hugging Face provides a guide on fine-tuning the Vision Transformer (ViT) model for image classification using the datasets and transformers libraries, demonstrating the process with the beans dataset.

  • Dispatch
    Vision Transformers on Hugging Face Optimum Graphcore

    Hugging Face and Graphcore enable the efficient fine-tuning of Vision Transformer (ViT) models on Intelligence Processing Units (IPUs) using the Optimum library, demonstrated through a multi-label chest X-ray classification task.

  • Project
    ZJU4HealthCare/HealthGPT

    A family of medical multimodal large language models that unify medical text, 2D images, and 3D volumes for advanced healthcare comprehension and generation.

  • Project
    kyegomez/MultiModalMamba

    A multi-modal AI model that integrates Vision Transformer (ViT) and Mamba to efficiently process and interpret text, images, audio, and video data simultaneously.