18 papers
CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight
Chensheng Peng, Wenhao Ding, Ran Tian +9
World action models (WAMs) jointly predict actions (intent) and visual future (foresight). Standard training adds noise to recorded actions and video simultaneously, but such train…
RoboSTAR: Next-Scale Autoregressive Sign Language Translation for Humanoid Robots
Yujia Zeng, Chensheng Peng, Yuxin Chen +3
Sign-language interpretation in public communication relies on qualified professional interpreters and can be difficult to scale, motivating robotic signing as a complementary acce…
Mitigating Compiler Fusion-Induced Power Bursts in Mobile NPU Inference as the Battery Depletes
Ryoga Yuzawa, Masayoshi Tomizuka
Mobile devices increasingly rely on real-time NPU inference for camera and perception workloads. Under low-voltage conditions, however, a single inference can induce an instantaneo…
ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation
Yu-Hsiang Chen, Wei-Jer Chang, Yi-Ting Chen +1
Controllable traffic simulation is critical for testing autonomous driving systems, yet existing approaches often require retraining large generative models with extensive annotate…
Scaling Nonlinear Optimization: Many Problems One GPU
John Viljoen, Johanna Haffner, Masayoshi Tomizuka +1
Many robotics problems, including trajectory optimization, inverse kinematics, and contact-rich motion planning, reduce to nonlinear programs (NLPs). Mature NLP solvers such as IPO…
TEXEDO : Test Time Scaling for Controller-aware Language-conditioned Humanoid Motion Generation
Jianuo Cao, Yuxin Chen, Yuzhen Song +3
Text-conditioned motion generation is a promising interface for programming humanoid robots, yet current generators are often trained on human motion datasets retargeted to robot m…
Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
Jiuming Liu, Chaojun Ni, Mengmeng Liu +7
With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, benefiting various downstream do…
CLAW: Composable Language-Annotated Whole-body Motion Generation
Jianuo Cao, Yuxin Chen, Masayoshi Tomizuka
Training language-conditioned whole-body controllers for humanoid robots demands large-scale motion-language datasets. Existing approaches based on motion capture are costly and li…
Rethinking Image-to-3D Generation with Sparse Queries: Efficiency, Capacity, and Input-View Bias
Zhiyuan Xu, Jiuming Liu, Yuxin Chen +3
We present SparseGen, a novel framework for efficient image-to-3D generation, which exhibits low input-view bias while being significantly faster. Unlike traditional approaches tha…
DiscreteRTC: Discrete Diffusion Policies are Natural Asynchronous Executors
Pengcheng Wang, Kaiwen Hong, Chensheng Peng +4
Unlike chatbots, physical AI must act while the world keeps evolving. Therefore, the inter-chunk pause of synchronous executors are fatal for dynamic tasks regardless of how fast t…
URoPE: Universal Relative Position Embedding across Geometric Spaces
Yichen Xie, Depu Meng, Chensheng Peng +4
Relative position embedding has become a standard mechanism for encoding positional information in Transformers. However, existing formulations are typically limited to a fixed geo…
Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning
Yichen Xie, Yixiao Wang, Shuqi Zhao +4
The generalization ability of imitation learning policies for robotic manipulation is fundamentally constrained by the diversity of expert demonstrations, while collecting demonstr…
RT-GS: Gaussian Splatting with Reflection and Transmittance Primitives
Kunnong Zeng, Chensheng Peng, Yichen Xie +2
Gaussian Splatting is a powerful tool for reconstructing diffuse scenes, but it struggles to simultaneously model specular reflections and the appearance of objects behind semi-tra…
RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space
Yichen Xie, Chensheng Peng, Mazen Abdelfattah +6
World foundation models aim to simulate the evolution of the real world with physically plausible behavior. Unlike prior methods that handle spatial and temporal correlations separ…
Adaptive Linear Path Model-Based Diffusion
Yutaka Shimizu, Masayoshi Tomizuka
The interest in combining model-based control approaches with diffusion models has been growing. Although we have seen many impressive robotic control results in difficult tasks, t…
StreamDiT: Real-Time Streaming Text-to-Video Generation
Akio Kodaira, Tingbo Hou, Ji Hou +4
Recently, great progress has been achieved in text-to-video (T2V) generation by scaling transformer-based diffusion models to billions of parameters, which can generate high-qualit…
Bisimulation metric for Model Predictive Control
Yutaka Shimizu, Masayoshi Tomizuka
Model-based reinforcement learning has shown promise for improving sample efficiency and decision-making in complex environments. However, existing methods face challenges in train…
Differentiable Predictive Control for Robotics: A Data-Driven Predictive Safety Filter Approach
John Viljoen, Wenceslao Shaw Cortez, Jan Drgona +3
Model Predictive Control (MPC) is effective at generating safe control strategies in constrained scenarios, at the cost of computational complexity. This is especially the case in…