SamuelSchmidgall/AgentClinic

What it solves

AgentClinic provides a way to evaluate AI models in simulated clinical environments. Instead of static tests, it uses language agents to simulate the dynamic interaction between a doctor and a patient to see how AI performs in a medical diagnostic process.

How it works

The system uses multiple LLM-based agents to create a simulation: a doctor agent and a patient agent interact with each other. It can simulate various clinical scenarios using datasets like MedQA, NEJM, and MIMIC-IV. The framework allows users to test different models (such as GPT-4o, Claude 3.5, or Llama 3) and can introduce specific biases (e.g., gender, race, or self-diagnosis) to see how they affect the clinical outcome.

Who it’s for

Researchers and developers building medical AI models who need a realistic, multimodal benchmark to test diagnostic accuracy and the impact of clinical biases.

Highlights

• Multimodal Support: Supports vision models and image requests within clinical cases.
• Diverse Datasets: Includes cases based on real clinical data from MIMIC-IV, NEJM, and MedQA.
• Bias Testing: Ability to inject specific doctor and patient biases (such as socioeconomic or cultural biases) into the simulation.
• Flexible Model Integration: Compatible with OpenAI, Replicate, and HuggingFace models.