ZJU4HealthCare/HealthGPT

【ICML 2025 Spotlight】 Official Repo for Paper ‘’HealthGPT : A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation‘’

What it solves

HealthGPT is a family of medical multimodal large language models (MLLMs) designed to bridge the gap between general-purpose AI and the specialized needs of healthcare. It addresses the challenge of processing diverse medical data types—such as 2D images, 3D volumes, and text—within a single framework for both understanding (comprehension) and generation.

How it works

The series consists of two primary models:

  • HealthGPT-Pro: A high-performance model built on Qwen3-VL. It uses a two-stage training recipe (3M alignment samples and 10M supervised fine-tuning samples) to analyze medical text, 2D images, and 3D volumetric data.
  • HealthGPT: A unified framework that uses heterogeneous low-rank adaptation (H-LoRA) and a three-stage learning strategy to enable a pre-trained LLM to handle both visual comprehension and generation tasks.

Who it’s for

This project is intended for medical AI researchers, healthcare data analysts, and developers building specialized medical diagnostic or generative tools.

Highlights

  • Broad Modality Support: Supports 14 different medical modalities, including CT, MRI, X-ray, Ultrasound, and Histopathology.
  • Unified Framework: Combines comprehension and generation capabilities in one system.
  • 3D Volume Analysis: Specifically supports 3D medical volumes, a critical requirement for advanced radiology.
  • State-of-the-Art Performance: Outperforms several existing medical MLLMs across multiple text and multimodal benchmarks.

Related

  • Project
    Warma10032/cyber-doctor

    A multimodal AI health assistant that combines RAG, knowledge graphs, and voice interaction to provide basic disease diagnosis and medical record analysis.

  • Project
    HUANGLIZI/LViT

    LViT is a multimodal framework that combines language and vision transformers to improve the accuracy of medical image segmentation in CT scans and other medical imagery.

  • Dispatch
    Hugging Face Vision Language Models 2025 Update

    Hugging Face provides a comprehensive overview of the 2024-2025 evolution of Vision Language Models (VLMs), highlighting trends in any-to-any architectures, reasoning models, and the rise of Vision-Language-Action (VLA) models for robotics.

  • Project
    OpenEnvision/Awesome-Multimodal-Modeling

    Awesome‑Multimodal‑Modeling is a community‑curated “awesome‑list” that surveys multimodal AI models. It defines four clear families—Traditional, MLLMs, Unified, and Native—explains their architectural differences, and links to open‑source model collections on Hugging Face. The repo serves researchers and engineers who need a structured view of the rapidly evolving multimodal landscape.

  • Dispatch
    MedGemma and MedSigLIP Release Notes

    Google DeepMind has released MedGemma and MedSigLIP, a collection of open multimodal models designed to provide a high-performance, privacy-preserving foundation for healthcare and lifesciences AI development.