6 papers
JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models
Zikun Zhang, Jiayuan Sheng, David D. Yao +1
On-policy distillation (OPD) is a post-training approach for large language models (LLMs) that enables a student model to learn a teacher model's distribution from sequences sample…
DOHF: Online Diffusion Fine-tuning with Doob's -transform Guidance
Zhengyi Guo, Jiayuan Sheng, Wenpin Tang +1
Reward-based diffusion fine-tuning faces practical challenges when desirable outcomes are rare or conditioning corrections are costly to estimate. In this work, we propose Diffusio…
Spec2Twin-Chain: Orchestrating Bi-Level Optimization with LLMs for Blockchain Digital Twin Construction
Haoting Zhang, Haoxian Chen, Jiayuan Sheng +4
Building a blockchain digital twin largely requires translating domain knowledge and specific system descriptions into a simulator architecture, calibrating its parameters against…
A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models
Zikun Zhang, Jiayuan Sheng, David D. Yao +1
We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces via a stochastic control approach, where the state dynam…
Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline
Zhengyi Guo, Jiayuan Sheng, David D. Yao +1
We propose a deterministic adjoint matching framework that formulates human preference alignment for flow-based generative models as an optimal control problem over velocity fields…
Understanding Sampler Stochasticity in Training Diffusion Models for RLHF
Jiayuan Sheng, Hanyang Zhao, Haoxian Chen +2
Reinforcement Learning from Human Feedback (RLHF) is increasingly used to fine-tune diffusion models, but a key challenge arises from the mismatch between stochastic samplers used…