16 papers
Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generation
Weijie Wang, Haomin Zhang, Ye Li +7
Long-form agentic video generation must turn a screenplay into shots that remain spatially and temporally consistent across nonconsecutive requests. Passing prompts or recent frame…
Provable Benefits of Regularization: Fast Rates for Adversarial Imitation Learning
Hanbin Zhou, Shangzhe Li, Alexander Braverman +1
We study adversarial imitation learning (AIL), in which an agent learns to imitate expert demonstrations by optimizing a policy against an adversarial reward that distinguishes exp…
An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
Shangzhe Li, Yuxiao Yang, Tianrun Yu +4
We study on-policy distillation (OPD) through the lens of reinforcement learning, establishing a connection between the reverse-KL objective in OPD and KL-regularized policy optimi…
Rethinking Training-Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct It
Tianrun Yu, Kaixiang Zhao, Shangzhe Li +4
We study training-inference mismatch in reinforcement learning with verifiable rewards (RLVR) for large language models, where rollouts are sampled by an inference engine while gra…
When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation
Yuxiao Yang, Tianrun Yu, Shangzhe Li +6
We study length inflation in on-policy distillation (OPD), where student responses can become excessively long and even exhaust the generation budget. We identify \emph{termination…
WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation
Baiqi Li, Ce Zhang, Yu Fang +4
A robot working alongside people must reason about what they have done, in what order, and with what intent. Video carries the spatial layouts, object histories, and gestures that…
Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation
Shangzhe Li, Weitong Zhang
We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained -function, the learner…
Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
Shangzhe Li, Xuchao Zhang, Chetan Bansal +1
Self-play post-training methods has emerged as an effective approach for finetuning large language models and turn the weak language model into strong language model without prefer…
Imitation from Observations with Trajectory-Level Generative Embeddings
Yongtao Qu, Shangzhe Li, Weitong Zhang
We consider the offline imitation learning from observations (LfO) where the expert demonstrations are scarce and the available offline suboptimal data are far from the expert beha…
Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
Xinming Gao, Shangzhe Li, Yujin Cai +1
Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly doma…
A Recipe for Efficient Sim-to-Real Transfer in Manipulation with Online Imitation-Pretrained World Models
Yilin Wang, Shangzhe Li, Haoyi Niu +4
We are interested in solving the problem of imitation learning with a limited amount of real-world expert data. Existing offline imitation methods often struggle with poor data cov…
Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learning
Shangzhe Li, Dongruo Zhou, Weitong Zhang
We study online adversarial imitation learning (AIL), where an agent learns from offline expert demonstrations and interacts with the environment online without access to rewards.…
Language Model Distillation: A Temporal Difference Imitation Learning Perspective
Zishun Yu, Shangzhe Li, Xinhua Zhang
Large language models have led to significant progress across many NLP tasks, although their massive sizes often incur substantial computational costs. Distillation has become a co…
Coupled Distributional Random Expert Distillation for World Model Online Imitation Learning
Shangzhe Li, Zhiao Huang, Hao Su
Imitation Learning (IL) has achieved remarkable success across various domains, including robotics, autonomous driving, and healthcare, by enabling agents to learn complex behavior…
Reward-free World Models for Online Imitation Learning
Shangzhe Li, Zhiao Huang, Hao Su
Imitation learning (IL) enables agents to acquire skills directly from expert demonstrations, providing a compelling alternative to reinforcement learning. However, prior online IL…
Augmenting Offline Reinforcement Learning with State-only Interactions
Shangzhe Li, Xinhua Zhang
Batch offline data have been shown considerably beneficial for reinforcement learning. Their benefit is further amplified by upsampling with generative models. In this paper, we co…