zeroQiaoba/AffectGPT

What it solves

This project addresses the limitations of traditional emotion recognition systems, which are typically restricted to a fixed, predefined set of emotion labels. By moving toward Open-Vocabulary Multimodal Emotion Recognition (OV-MER), it enables models to understand and predict a broader, more nuanced spectrum of human emotions using generative AI.

How it works

The project provides a suite of frameworks and datasets that shift emotion recognition from simple classification to generative understanding. It uses Multimodal Large Language Models (MLLMs) to interpret emotional content. To overcome the misalignment between standard token-level training and human-centric emotion metrics, it introduces reinforcement learning (AffectGPT-RL). This allows the model to optimize directly against non-differentiable, emotion-specific evaluation metrics (like emotion wheels) that standard gradient-based training cannot handle.

Who it’s for

This project is designed for researchers and developers in affective computing and multimodal AI who want to build systems capable of fine-grained, open-ended emotion recognition rather than basic category classification.

Highlights

• OV-MER: A framework that removes label space restrictions for more flexible emotion prediction.
• EmoPrefer: A dataset and benchmark for evaluating how well LLMs understand human emotion preferences.
• AffectGPT-RL: A reinforcement learning approach that aligns model training with human-centric emotion evaluation metrics.
• Comprehensive Datasets: Includes multiple research-grade datasets (MER-Caption, EmoPrefer-Data) for training and benchmarking.