10 papers
OPD+: Rethinking the Advantage Design for On-Policy Distillation
Hanyang Zhao, Haoxian Chen, Han Lin +3
On-policy distillation (OPD) is a widely used technique to transfer capabilities from capable teacher language models to the base student models, and can be formulated in a reinfor…
DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
Hanyang Zhao, Dawen Liang, Wenpin Tang +2
We propose DiFFPO, Diffusion Fast and Furious Policy Optimization, a unified framework for training masked diffusion large language models (dLLMs) to reason not only better (furiou…
RPO: Fine-Tuning Visual Generative Models via Rich Vision-Language Preferences
Hanyang Zhao, Haoxian Chen, Yucheng Guo +5
Traditional preference tuning methods for LLMs/Visual Generative Models often rely solely on reward model labeling, which can be opaque, offer limited insights into the rationale b…
Understanding Sampler Stochasticity in Training Diffusion Models for RLHF
Jiayuan Sheng, Hanyang Zhao, Haoxian Chen +2
Reinforcement Learning from Human Feedback (RLHF) is increasingly used to fine-tune diffusion models, but a key challenge arises from the mismatch between stochastic samplers used…
SOCRATES: Simulation Optimization with Correlated Replicas and Adaptive Trajectory Evaluations
Haoting Zhang, Haoxian Chen, Donglin Zhan +5
The field of simulation optimization (SO) encompasses various methods developed to optimize complex, expensive-to-sample stochastic systems. Established methods include, but are no…
Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning
Hanyang Zhao, Haoxian Chen, Ji Zhang +2
Reinforcement learning from human feedback (RLHF), which aligns a diffusion model with input prompt, has become a crucial step in building reliable generative AI models. Most works…