17 papers
How Should Vision-Language-Action Models Use Proprioceptive State?
Yiren Zhao, Ziyang Chen, Ziyang Rao +5
Recent Vision-Language-Action (VLA) models almost universally take robot proprioceptive state as input, yet wire it in incompatible ways -- serialized into text prompts, projected…
Source-Lifted Flow Matching for Intervenable Multimodal Imitation
He Zhang, Ying Sun, Pengteng Li +7
Flow-matching policies are promising for imitation learning because they model complex multimodal action distributions. However, their stochasticity is largely passive: repeated sa…
PHASER: Phase-Aware and Semantic Experience Replay for Vision-Language-Action Models
Ziyang Chen, Shaoguang Wang, Weiyu Guo +5
Vision-Language-Action (VLA) models have achieved remarkable success in language-conditioned robotic manipulation. However, deploying these models in open-ended environments requir…
Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action
Pengteng Li, Weiyu Guo, He Zhang +4
We introduce SOMA, the Spatial Memory framework for Out-of-Vision Manipulation in Vision-Language-Action (VLA) models. Most existing VLAs implicitly assume that task-relevant objec…
NeuroVLA: A Brain-inspired Embodied Intelligence for Fluid and Fast Reflexive Robotics Control
Weiyu Guo, He Zhang, Pengteng Li +7
Vision-language-action (VLA) models inherit powerful semantic representations from large language models, but carrying over their design priorities can leave the demands of physica…
From Events to Clarity: The Event-Guided Diffusion Framework for Dehazing
Ling Wang, Yunfan Lu, Wenzong Ma +3
Clear imaging under hazy conditions is a critical task. Prior-based and neural methods have improved results. However, they operate on RGB frames, which suffer from limited dynamic…
Beyond Boundaries: Leveraging Vision Foundation Models for Source-Free Object Detection
Huizai Yao, Sicheng Zhao, Pengteng Li +6
Source-Free Object Detection (SFOD) aims to adapt a source-pretrained object detector to a target domain without access to source data. However, existing SFOD methods predominantly…
You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMs
Yijie Xu, Huizai Yao, Zhiyu Guo +5
Large language models (LLMs) are increasingly deployed in specialized domains such as finance, medicine, and agriculture, where they face significant distribution shifts from their…
See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model
Pengteng Li, Pinhao Song, Wuyang Li +5
We introduce SEE&TREK, the first training-free prompting framework tailored to enhance the spatial understanding of Multimodal Large Language Models (MLLMS) under vision-only const…
DeblurSplat: SfM-free 3D Gaussian Splatting with Event Camera for Robust Deblurring
Pengteng Li, Yunfan Lu, Pinhao Song +4
In this paper, we propose the first Structure-from-Motion (SfM)-free deblurring 3D Gaussian Splatting method via event camera, dubbed DeblurSplat. We address the motion-deblurring…
Equivariant Volumetric Grasping
Pinhao Song, Yutong Hu, Pengteng Li +1
We propose a new volumetric grasp model that is equivariant to rotations around the vertical axis, leading to a significant improvement in sampling efficiency. Our model employs a…
From Events to Enhancement: A Survey on Event-Based Imaging Technologies
Yunfan Lu, Xiaogang Xu, Pengteng Li +4
Event cameras offering high dynamic range and low latency have emerged as disruptive technologies in imaging. Despite growing research on leveraging these benefits for different im…
Multimodal 3D Genome Pre-training
Minghao Yang, Pengteng Li, Yan Liang +6
Deep learning techniques have driven significant progress in various analytical tasks within 3D genomics in computational biology. However, a holistic understanding of 3D genomics…
SEE: See Everything Every Time -- Adaptive Brightness Adjustment for Broad Light Range Images via Events
Yunfan Lu, Xiaogang Xu, Hao Lu +8
Event cameras, with a high dynamic range exceeding , significantly outperform traditional embedded cameras, robustly recording detailed changing information under various li…
EventVL: Understand Event Streams via Multimodal Large Language Model
Pengteng Li, Yunfan Lu, Pinghao Song +3
The event-based Vision-Language Model (VLM) recently has made good progress for practical vision tasks. However, most of these works just utilize CLIP for focusing on traditional p…
HR-INR: Continuous Space-Time Video Super-Resolution via Event Camera
Yunfan Lu, Yusheng Wang, Zipeng Wang +3
Continuous space-time video super-resolution (C-STVSR) aims to simultaneously enhance video resolution and frame rate at an arbitrary scale. Recently, implicit neural representatio…
Robot Trajectron: Trajectory Prediction-based Shared Control for Robot Manipulation
Pinhao Song, Pengteng Li, Erwin Aertbelien +1
We address the problem of (a) predicting the trajectory of an arm reaching motion, based on a few seconds of the motion's onset, and (b) leveraging this predictor to facilitate sha…