yongliang-wu/DFT

[ICLR 2026] On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification.

What it solves

Standard Supervised Fine-Tuning (SFT) often suffers from limited generalization compared to Reinforcement Learning (RL). This project implements Dynamic Fine-Tuning (DFT), a method designed to improve the generalization capabilities of Large Language Models (LLMs) by addressing a problematic reward structure implicitly encoded in standard SFT gradients.

How it works

DFT is a simple modification to the SFT objective function. It stabilizes gradient updates for each token by dynamically rescaling the loss of each token by its predicted probability (which is detached from the gradient flow to prevent backpropagation through the probability itself).

Who it’s for

Researchers and developers working on LLM fine-tuning, specifically those focusing on mathematical reasoning, complex coding problems, and multimodal reasoning where multiple valid reasoning paths (non-deterministic trajectories) exist.

Highlights

  • One-line implementation: The core logic is a single line of code change to the standard SFT loss calculation.
  • Improved Generalization: Outperforms standard SFT across multiple challenging benchmarks and base models.
  • RL-like benefits: Offers an effective and simpler alternative to offline RL settings.
  • Broad Integration: Already supported by popular libraries like Hugging Face TRL, LLaMA-Factory, and ms-swift.

Related

  • Project
    agentscope-ai/Trinity-RFT

    A general-purpose framework for reinforcement fine-tuning of LLMs that decouples exploration, training, and data buffering to support diverse RL algorithms and agentic workflows.

  • Project
    dllm-reasoning/d1

    d1 is a framework for scaling reasoning in diffusion Large Language Models using a combination of masked Supervised Fine-Tuning and a novel reinforcement learning method called diffu-GRPO.

  • Dispatch
    Self-Distillation: A New Frontier for Continual Learning in LLMs

    Exploring how self-distillation transforms off-policy data into on-policy data to mitigate catastrophic forgetting during LLM fine-tuning.

  • Project
    X-GenGroup/Flow-Factory

    A unified framework for online RL and offline fine-tuning of diffusion and flow-matching models across image, video, and audio-video modalities.

  • Dispatch
    Fine-tuning Llama 2 with Direct Preference Optimization (DPO) via TRL

    Hugging Face has integrated Direct Preference Optimization (DPO) into the TRL library, enabling the alignment of Llama 2 and other LLMs using a simplified binary cross-entropy loss instead of complex RLHF pipelines.