ZJUI-AI4H/Hulu-Med

A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding

📚 What is Hulu‑Med?

Hulu‑Med is an open‑source, multimodal medical vision‑language model (VLM) that can understand and generate text about medical text, 2‑D images, 3‑D volumes (e.g., CT/MRI NIfTI files), and videos. The authors trained the model on 16.7 M public medical samples covering 12 anatomical systems and 14 imaging modalities, and they release the full training pipeline, data curation scripts, and model weights.

🎯 Core capabilities

  • Holistic multimodal reasoning – answer questions, write reports, or perform diagnosis‑style reasoning on any combination of text, images, 3‑D scans, and surgical videos.
  • Transparent & reproducible – all code, data lists, and weights are publicly available under the Apache‑2.0 license.
  • Efficient inference – compatible with HuggingFace Transformers, vLLM, flash‑attention, and tensor‑parallelism for fast serving.
  • Multiple model sizes – from a lightweight 4 B‑parameter model up to a 235 B‑parameter “Flash‑Preview” variant, letting users pick the right trade‑off between speed, memory, and accuracy.

📦 Model zoo (selected variants)

Model Params Base LLM Approx. GPU hours* HF link
Hulu‑Med‑4B 4 B Qwen3‑VL‑4B ~1 k 🤗
Hulu‑Med‑7B 7 B Qwen2.5‑7B ~4 k 🤗
Hulu‑Med‑14B 14 B Qwen3‑14B ~8 k 🤗
Hulu‑Med‑30A3 30 B Qwen3‑VL‑30A3B ~3.2 k 🤗
Hulu‑Med‑235A22 (Flash‑Preview) 235 B Qwen3‑VL‑235A22B ~10 k 🤗

*GPU‑hours are rough estimates for a single‑node training run.

📈 How well does it perform?

The authors benchmarked on 30 medical VLM and text datasets (e.g., VQA‑RAD, PathVQA, MedXQA, MMLU‑Pro). Across the board, Hulu‑Med beats existing open‑source medical VLMs and often narrows the gap to proprietary models like GPT‑4.0. Highlights:

  • Hulu‑Med‑7B reaches 84.2 on OM.VQA and 66.8 on MedXQA, the best among <10 B open models.
  • Hulu‑Med‑235A22 hits 70.5 on VQA‑RAD and 39.8 on MedXQA, surpassing all listed competitors.
  • On pure‑text medical exams (MMLU‑Pro, Medbullets, etc.) the 14 B and 30 B variants achieve 68–69 points, the top scores among open VLMs.

🛠️ Quick start (HuggingFace Transformers)

from transformers import AutoModelForCausalLM, AutoProcessor
import torch

model_id = "ZJU-AI4H/Hulu-Med-7B"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
    attn_implementation="flash_attention_2",
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Example: generate a report from a 2‑D X‑ray image
conversation = [{
    "role": "user",
    "content": [
        {"type": "image", "image": {"image_path": "./demo/xray.jpg"}},
        {"type": "text", "text": "Write a concise radiology report."},
    ]
}]
inputs = processor(conversation=conversation, return_tensors="pt", add_generation_prompt=True)
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
output_ids = model.generate(**inputs, max_new_tokens=1024)
print(processor.batch_decode(output_ids, skip_special_tokens=True)[0])

The same processor works for 3‑D NIfTI volumes (type: "3d") and videos (type: "video"). See the README for detailed examples.

⚡ Running with vLLM (high‑throughput serving)

VLLM_USE_PRECOMPILED=1 pip install git+https://github.com/jiangsongtao/vllm.git
pip install decord ffmpeg imageio   # video support

After installation, launch vLLM with the model path and use the standard OpenAI‑compatible API to send multimodal payloads.

📂 Repository layout (high‑level)

  • scripts/ – data‑curation, preprocessing, and training launch scripts.
  • hulu_med/ – model definition and custom processor that parses the multimodal conversation format.
  • demo/ – example images, videos, and 3‑D scans used in the README.
  • requirements.txt – full Python dependency list.
  • LICENSE – Apache 2.0.

📜 License & citation

The code and weights are released under Apache 2.0. If you use the models in research or products, cite the arXiv paper linked in the badge (arXiv:2510.08668).

🔗 Useful links

🤔 When to use Hulu‑Med?

  • Clinical AI research that needs a publicly available VLM for medical imaging or video.
  • Educational tools that generate explanations or reports from radiology images.
  • Prototype building where you want a single model that can handle text, 2‑D, 3‑D, and video inputs without stitching together separate systems.

🚀 Next steps

  1. Pick a model size that fits your GPU budget.
  2. Follow the installation steps (conda + pip) or the vLLM route for large‑scale serving.
  3. Use the provided Python snippets to start experimenting with text, image, 3‑D, or video inputs.
  4. Contribute back by reporting bugs, adding new public medical datasets, or fine‑tuning on a specialized sub‑domain.

Related