StanfordMIMI/Merlin

What it solves

Merlin addresses the need for a specialized foundation model for computed tomography (CT) scans that can integrate diverse medical data sources, such as 3D imaging, structured electronic health records (EHR), and unstructured radiology reports, to perform complex medical analysis.

How it works

It is a 3D Vision-Language Model (VLM) pretrained on both structured EHR data and unstructured radiology reports. The model can be configured for different specific tasks by enabling different components during initialization, such as image embeddings, phenotype classification, disease prediction, or report generation.

Who it’s for

Medical AI researchers and clinicians working with CT imaging, radiology report automation, and predictive healthcare analytics.

Highlights

• Supports 3D CT scan analysis.
• Capable of generating radiology reports.
• Performs phenotype classification and five-year disease prediction.
• Includes a released Abdominal CT Dataset for community use.
• Integrates with the nnU-Net framework for segmentation tasks.