shibing624/MedicalGPT

What it solves

MedicalGPT provides a comprehensive training pipeline for creating domain-specific large language models, specifically tailored for the medical industry. It simplifies the process of transforming a general-purpose LLM into a specialized medical assistant by providing the tools for every stage of the training lifecycle, from initial knowledge injection to human preference alignment.

How it works

The project implements a multi-stage training pipeline based on the ChatGPT training methodology:

1. Continue Pre-training (PT): Injects domain-specific knowledge by training on massive medical documents.
2. Supervised Fine-tuning (SFT): Aligns the model with instruction-following behavior using curated question-and-answer pairs.
3. Alignment/Preference Optimization: Refines model behavior using several available methods:
   • RLHF: A two-step process involving Reward Modeling (RM) and Reinforcement Learning (PPO/RLOO).
   • DPO (Direct Preference Optimization): Directly optimizes the model based on human preferences without a separate reward model.
   • ORPO: Combines SFT and alignment into a single step to reduce catastrophic forgetting.
   • GRPO: Uses rule-based rewards to train reasoning chains.
   • OPD (On-Policy Distillation): Distills knowledge from a stronger teacher model.

Additionally, the project supports Agent Finetuning, allowing models to learn function calling (Tool Use) by introducing specific roles like function_call and observation into the conversation templates.

Who it’s for

• AI Researchers and Developers looking to build specialized medical LLMs.
• Developers wanting to implement advanced alignment techniques like DPO, GRPO, or RLHF on their own datasets.
• Organizations needing a medical-domain model that can interact with external tools via function calling.

Highlights

• Full-lifecycle pipeline: Covers PT, SFT, RLHF, DPO, ORPO, GRPO, and OPD.
• Broad model support: Compatible with Llama-3, Qwen-2.5, Mixtral, and others.
• Agent capabilities: Built-in support for training models to perform function calls.
• Flexible training: Supports full-parameter tuning, LoRA, and QLoRA to accommodate different hardware constraints.
• RAG integration: Includes a ChatPDF demo for retrieval-augmented generation based on knowledge base files.