Publications (63)
Defeating the Training-Inference Mismatch via FP16
Penghui Qi, Zichen Liu, Xiangxin Zhou +4
Reinforcement learning (RL) fine-tuning of large language models (LLMs) often suffers from instability due to the numerical mismatch between the training and inference policies. Wh…
Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
Han Lu, Zichen Liu, Shaopan Xiong +19
Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing Large Language Models (LLMs) with diverse capabilities. However, many systems desi…
MagicQuill: An Intelligent Interactive Image Editing System
Zichen Liu, Yue Yu, Hao Ouyang +6
Image editing involves a variety of complex tasks and requires efficient and precise manipulation techniques. In this paper, we present MagicQuill, an integrated image editing syst…
Optimizing Anytime Reasoning via Budget Relative Policy Optimization
Penghui Qi, Zichen Liu, Tianyu Pang +3
Scaling test-time compute is crucial for enhancing the reasoning capabilities of large language models (LLMs). Existing approaches typically employ reinforcement learning (RL) to m…
State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding
Jiahuan Zhou, Kai Zhu, Zhenyu Cui +3
Recently, pre-trained state space models have shown great potential for video classification, which sequentially compresses visual tokens in videos with linear complexity, thereby…
RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
Wei Gao, Yuheng Zhao, Tianyuan Wu +15
Agentic Reinforcement Learning (RL) trains LLMs through multi-turn interactions with environments, producing workloads that mix compute-bound prefill, bandwidth-bound decoding, CPU…
SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis
Zijian Wu, Jinjie Ni, Xiangyan Liu +3
Vision-language models (VLMs) trained via reinforcement learning with verifiable reward (RLVR) have shown notable progress in scaling test-time compute effectively. In this work, w…
AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes
Rao Fu, Zehao Wen, Zichen Liu +1
Inspired by cognitive theories, we introduce AnyHome, a framework that translates any text into well-structured and textured indoor scenes at a house-scale. By prompting Large Lang…
Rethinking the Trust Region in LLM Reinforcement Learning
Penghui Qi, Xiangxin Zhou, Zichen Liu +4
Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorith…
Locality Sensitive Sparse Encoding for Learning World Models Online
Zichen Liu, Chao Du, Wee Sun Lee +1
Acquiring an accurate world model online for model-based reinforcement learning (MBRL) is challenging due to data nonstationarity, which typically causes catastrophic forgetting fo…
Selective Visual Prompting in Vision Mamba
Yifeng Yao, Zichen Liu, Zhenyu Cui +2
Pre-trained Vision Mamba (Vim) models have demonstrated exceptional performance across various computer vision tasks in a computationally efficient manner, attributed to their uniq…
Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queries
Yuhao Wang, Wenjie Qu, Shengfang Zhai +5
Retrieval-Augmented Generation (RAG) systems enhance large language models (LLMs) by incorporating external knowledge bases, but this may expose them to extraction attacks, leading…
EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine
Jiayi Weng, Min Lin, Shengyi Huang +9
There has been significant progress in developing reinforcement learning (RL) training systems. Past works such as IMPALA, Apex, Seed RL, Sample Factory, and others, aim to improve…
Superconductivity in ThMo2Si2C with Mo2C Square Net
Zichen Liu, Baizhuo Li, Yusen Xiao +16
We report the superconductivity of a new quaternary compound ThMoSiC, synthesized with the arc-melting technique. The compound crystallizes in a tetragonal CeCrSiC-…
Class-aware Domain Knowledge Fusion and Fission for Continual Test-Time Adaptation
Jiahuan Zhou, Chao Zhu, Zhenyu Cui +3
Continual Test-Time Adaptation (CTTA) aims to quickly fine-tune the model during the test phase so that it can adapt to multiple unknown downstream domain distributions without pre…
Understanding R1-Zero-Like Training: A Critical Perspective
Zichen Liu, Changyu Chen, Wenjun Li +5
DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critic…
MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research
Dingbang Wu, Rui Hao, Haiyang Wang +8
We present MobileGym, a browser-hosted, lightweight, fully controllable environment for everyday mobile use, targeting interaction fidelity without replicating proprietary backends…
Token Coordinated Prompt Attention is Needed for Visual Prompting
Zichen Liu, Xu Zou, Gang Hua +1
Visual prompting techniques are widely used to efficiently fine-tune pretrained Vision Transformers (ViT) by learning a small set of shared prompts for all tokens. However, existin…
Sample-Efficient Alignment for LLMs
Zichen Liu, Changyu Chen, Chao Du +2
We study methods for efficiently aligning large language models (LLMs) with human preferences given budgeted online feedback. We first formulate the LLM alignment problem in the fr…
Breaking the Impasse: Dual-Scale Evolutionary Policy Training for Social Language Agents
Minzheng Wang, Run Luo, Yanbo Wang +6
While Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for closed-ended tasks, extending it to open-ended social language games via self-play reveals a cr…
Memory Reviver: Supporting Photo-Collection Reminiscence for People with Visual Impairment via a Proactive Chatbot
Shuchang Xu, Chang Chen, Zichen Liu +4
Reminiscing with photo collections offers significant psychological benefits but poses challenges for people with visual impairment (PVI). Their current reliance on sighted help re…
Riemannian Denoising Diffusion Probabilistic Models
Zichen Liu, Wei Zhang, Christof Schütte +1
We propose Riemannian Denoising Diffusion Probabilistic Models (RDDPMs) for learning distributions on submanifolds of Euclidean space that are level sets of functions, including mo…
From to : Investigating Reinforcement Learning in Pre-train Space
Yuqiao Tan, Minzheng Wang, Bo Liu +5
While reinforcement learning with verifiable rewards (RLVR) significantly enhances LLM reasoning by optimizing the conditional distribution P(y|x), its potential is fundamentally b…
GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model
Zixiang Ai, Zichen Liu, Yuanhang Lei +3
Pre-trained 3D vision models have gained significant attention for their promising performance on point cloud data. However, fully fine-tuning these models for downstream tasks is…
Reinforcing General Reasoning without Verifiers
Xiangxin Zhou, Zichen Liu, Anya Sims +6
The recent paradigm shift towards training large language models (LLMs) using DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has led to impressive advance…
Long-distance distribution of telecom time-energy entanglement generated on a silicon chip
Yuan-yuan Zhao, Fuyong Yue, Feng Gao +5
Entanglement distribution is a critical technique that enables numerous quantum applications. Most fiber-based long-distance experiments reported to date have utilized photon pair…
WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory
Hanlin Wang, Hao Ouyang, Qiuyu Wang +10
We present WorldDirector, a highly controllable video world model framework designed for persistent dynamic object memory and unrestricted viewpoint exploration. Unlike existing wo…
Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity
Zichen Liu, Wei Zhang, Tiejun Li
Euclidean diffusion models have achieved remarkable success in generative modeling across diverse domains, and they have been extended to manifold cases in recent advances. Instead…
Efficient Offline Policy Optimization with a Learned Model
Zichen Liu, Siyi Li, Wee Sun Lee +2
MuZero Unplugged presents a promising approach for offline policy learning from logged data. It conducts Monte-Carlo Tree Search (MCTS) with a learned model and leverages Reanalyze…
Gym-V: A Unified Vision Environment System for Agentic Vision Research
Fanqing Meng, Lingxiao Du, Jiawei Gu +9
As agentic systems increasingly rely on reinforcement learning from verifiable rewards, standardized ``gym'' infrastructure has become essential for rapid iteration, reproducibilit…
Learning Stateful Predictive Knowledge From Experience
Yan Song, Xidong Feng, Bo Liu +7
As large language model (LLM) agents increasingly learn from experience, they primarily rely on trajectory-level reflection to extract insights. Viewed through the lens of predicti…
Exploring the Potential of Integrated Optical Sensing and Communication (IOSAC) Systems with Si Waveguides for Future Networks
Xiangpeng Ou, Ying Qiu, Ming Luo +17
Advanced silicon photonic technologies enable integrated optical sensing and communication (IOSAC) in real time for the emerging application requirements of simultaneous sensing an…
CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narratives
Yihao Meng, Zichen Liu, Hao Ouyang +11
Autoregressive video generation aims at real-time, open-ended synthesis. Yet, cinematic storytelling is not merely the endless extension of a single scene; it requires progressing…
Distributed Learning for Principle Eigenspaces without Moment Constraints
Yong He, Zichen Liu, Yalin Wang
Distributed Principal Component Analysis (PCA) has been studied to deal with the case when data are stored across multiple machines and communication cost or privacy concerns prohi…
Precise Time Delay Measurement and Compensation for Tightly Coupled Underwater SINS/piUSBL Navigation
Jin Huang, Yingqiang Wang, Haoda Li +3
In multisensor systems, time synchronization is particularly challenging for underwater integrated navigation systems (INSs) incorporating acoustic positioning, where time delays c…
BinCtx: Multi-Modal Representation Learning for Robust Android App Behavior Detection
Zichen Liu, Shao Yang, Xusheng Xiao
Mobile app markets host millions of apps, yet undesired behaviors (e.g., disruptive ads, illegal redirection, payment deception) remain hard to catch because they often do not rely…
STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding
Zichen Liu, Kunlun Xu, Bing Su +3
Pre-trained on tremendous image-text pairs, vision-language models like CLIP have demonstrated promising zero-shot generalization across numerous image-based tasks. However, extend…
GRID: Graph Representation of Intelligence Data for Security Text Knowledge Graph Construction
Liangyi Huang, Zichen Liu, Fei Shao +5
Security knowledge graphs can provide computable external memory for security agents, but constructing them from long-form cyber threat intelligence (CTI) remains difficult: LLMs o…
An Improved ESO-Based Line-of-Sight Guidance Law for Path Following of Underactuated Autonomous Underwater Helicopter With Nonlinear Tracking Differentiator and Anti-saturation Controller
Haoda Li, Zichen Liu, Jin Huang +2
This paper presents an Improved Extended-state-observer based Line-of-Sight (IELOS) guidance law for path following of underactuated Autonomous Underwater helicopter (AUH) utilizin…
SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
Bo Liu, Leon Guertler, Simon Yu +9
Recent advances in reinforcement learning have shown that language models can develop sophisticated reasoning through training on tasks with verifiable rewards, but these approache…
Efficient Process Reward Model Training via Active Learning
Keyu Duan, Zichen Liu, Xin Mao +5
Process Reward Models (PRMs) provide step-level supervision to large language models (LLMs), but scaling up training data annotation remains challenging for both humans and LLMs. T…
Bootstrapping Language Models with DPO Implicit Rewards
Changyu Chen, Zichen Liu, Chao Du +5
Human alignment in large language models (LLMs) is an active area of research. A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the proces…
MagicQuillV2: Precise and Interactive Image Editing with Layered Visual Cues
Zichen Liu, Yue Yu, Hao Ouyang +11
We propose MagicQuill V2, a novel system that introduces a \textbf{layered composition} paradigm to generative image editing, bridging the gap between the semantic power of diffusi…
SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Prompting
Chenyu Zhang, Kunlun Xu, Zichen Liu +2
Vision-language models (VLMs) encounter considerable challenges when adapting to domain shifts stemming from changes in data distribution. Test-time adaptation (TTA) has emerged as…
Dynamic Typography: Bringing Text to Life via Video Diffusion Prior
Zichen Liu, Yihao Meng, Hao Ouyang +4
Text animation serves as an expressive medium, transforming static communication into dynamic experiences by infusing words with motion to evoke emotions, emphasize meanings, and c…
Infinite Worlds with Versatile Interactions
Zelin Gao, Qiuyu Wang, Jiapeng Zhu +17
We present LingBot-World 2.0 (also known as LingBot-World-Infinity), an advanced iteration of LingBot-World featuring four distinct upgrades. (1) Our model achieves an unbounded in…
GEM: A Gym for Agentic LLMs
Zichen Liu, Anya Sims, Keyu Duan +16
The training paradigm for large language models (LLMs) is moving from static datasets to experience-based learning, where agents acquire skills via interacting with complex environ…
Variational Reasoning for Language Models
Xiangxin Zhou, Zichen Liu, Haonan Wang +5
We introduce a variational reasoning framework for language models that treats thinking traces as latent variables and optimizes them through variational inference. Starting from t…
The Anatomy of Scam Scenarios: Large-Scale Characterization and Conversation-Aware Detection
Shang Ma, Chen Yanai, Avichai Ben +3
Online scams have become a pervasive global threat, causing substantial financial, psychological, and operational harm. Scammers embed psychological techniques (PTs) within reusabl…
RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback
Xiaoying Zhang, Zichen Liu, Yipeng Zhang +2
Standard reinforcement learning (RL) for large language model (LLM) agents primarily optimizes extrinsic task rewards, often favoring isolated task completion over continual adapta…
A DVL Aided Loosely Coupled Inertial Navigation Strategy for AUVs with Attitude Error Modeling and Variance Propagation
Jin Huang, Zichen Liu, Haoda Li +2
In underwater navigation systems, strap-down inertial navigation system/Doppler velocity log (SINS/DVL)-based loosely coupled architectures are widely adopted. Conventional approac…
Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset
Qingyan Bai, Qiuyu Wang, Hao Ouyang +10
Instruction-based video editing promises to democratize content creation, yet its progress is severely hampered by the scarcity of large-scale, high-quality training data. We intro…
Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
Weixun Wang, Shaopan Xiong, Gengru Chen +38
We introduce ROLL, an efficient, scalable, and user-friendly library designed for Reinforcement Learning Optimization for Large-scale Learning. ROLL caters to three primary user gr…
Vision Graph Prompting via Semantic Low-Rank Decomposition
Zixiang Ai, Zichen Liu, Jiahuan Zhou
Vision GNN (ViG) demonstrates superior performance by representing images as graph structures, providing a more natural way to capture irregular semantic patterns beyond traditiona…
Language Models Can Learn from Verbal Feedback Without Scalar Rewards
Renjie Luo, Zichen Liu, Xiangyan Liu +5
LLMs are often trained with RL from human or AI feedback, yet such methods typically compress nuanced feedback into scalar rewards, discarding much of their richness and inducing s…
FES-FM: Free Energy Surface Sampling via Reduced Flow Matching
Zichen Liu, Tiejun Li
Sampling the distribution of collective variables (CVs) and estimating the associated free energy surface are crucial problems in statistical physics, as they underpin a better und…
Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning
Tianyi Bai, Yuxuan Fan, Jiantao Qiu +7
Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks but still struggle with fine-grained visual differences, leading to hallucination…
Continual Reinforcement Learning by Planning with Online World Models
Zichen Liu, Guoji Fu, Chao Du +2
Continual reinforcement learning (CRL) refers to a naturalistic setting where an agent needs to endlessly evolve, by trial and error, to solve multiple tasks that are presented seq…
Adversarial Demonstration Attacks on Large Language Models
Jiongxiao Wang, Zichen Liu, Keun Hee Park +5
With the emergence of more powerful large language models (LLMs), such as ChatGPT and GPT-4, in-context learning (ICL) has gained significant prominence in leveraging these models…
TreeMAN: Tree-enhanced Multimodal Attention Network for ICD Coding
Zichen Liu, Xuyuan Liu, Yanlong Wen +3
ICD coding is designed to assign the disease codes to electronic health records (EHRs) upon discharge, which is crucial for billing and clinical statistics. In an attempt to improv…
Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs
Longxu Dou, Qian Liu, Fan Zhou +38
Sailor2 is a family of cutting-edge multilingual language models for South-East Asian (SEA) languages, available in 1B, 8B, and 20B sizes to suit diverse applications. Building on…
Calligrapher: Freestyle Text Image Customization
Yue Ma, Qingyan Bai, Hao Ouyang +8
We introduce Calligrapher, a novel diffusion-based framework that innovatively integrates advanced text customization with artistic typography for digital calligraphy and design ap…
Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
Shengyi Huang, Quentin Gallouédec, Florian Felten +30
In many Reinforcement Learning (RL) papers, learning curves are useful indicators to measure the effectiveness of RL algorithms. However, the complete raw data of the learning curv…