ZebangCheng/Emotion-LLaMA

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

What it solves

Emotion-LLaMA addresses the difficulty of capturing complex, real-world emotional expressions that are inherently multimodal. Traditional single-modality approaches often miss subtle cues, and existing Multimodal Large Language Models (MLLMs) frequently struggle to integrate audio data or recognize fine-grained facial micro-expressions.

How it works

The project integrates audio, visual, and textual inputs using a set of emotion-specific encoders (HuBERT for audio, EVA for global visual features, MAE for local visual features, and VideoMAE for temporal features). These features are aligned into a shared space and fed into a modified LLaMA model that has undergone instruction tuning. To support this, the authors introduced the MERR dataset, which provides both coarse-grained and fine-grained emotional annotations across diverse scenarios.

Who it’s for

Researchers and developers working on human-computer interaction, education, and counseling applications where accurate emotion perception and reasoning are required.

Highlights

  • Multimodal Integration: Combines audio, visual, and text to improve emotion recognition accuracy.
  • Instruction Tuned: Uses a specialized dataset (MERR) to enhance both recognition and reasoning capabilities.
  • ** uma-SOTA Performance**: Achieved top scores in the MER2023 and MER2024 challenges and outperformed other MLLMs on the EMER dataset.
  • Comprehensive Tooling: Includes a local demo, batch inference capabilities, and detailed API guides.

Related

  • Dispatch
    Hugging Face Vision Language Models 2025 Update

    Hugging Face provides a comprehensive overview of the 2024-2025 evolution of Vision Language Models (VLMs), highlighting trends in any-to-any architectures, reasoning models, and the rise of Vision-Language-Action (VLA) models for robotics.

  • Project
    bytedance/SALMONN

    A suite of advanced multi-modal LLMs that provide large language models with generic hearing abilities and integrated audio-visual perception.

  • Project
    s-kostyaev/ellama

    An LLM assistant for Emacs that provides chat, agentic coding workflows, and deep context integration across buffers and files.

  • Dispatch
    ConTextual: Benchmarking Multimodal Reasoning in Text-Rich Scenes

    Hugging Face and UCLA researchers have introduced ConTextual, a dataset and leaderboard designed to evaluate how Large Multimodal Models (LMMs) jointly reason over text and visual cues in complex, text-rich images.

  • Project
    MME-Benchmarks/Video-MME-v2

    A robust benchmark for evaluating video understanding in MLLMs, using a three-level progressive difficulty scale and grouped non-linear scoring to measure true temporal reasoning.