84 citations · 199 across the 23 of their papers we have counts for
24 papers
3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model
Hongyan Zhi, Peihao Chen, Siyuan Zhou +4
Manipulation has long been a challenging task for robots, while humans can effortlessly perform complex interactions with objects, such as hanging a cup on the mug rack. A key reas…
Learning 3D Persistent Embodied World Models
Siyuan Zhou, Yilun Du, Yuncong Yang +4
The ability to simulate the effects of future actions on the world is a crucial ability of intelligent embodied agents, enabling agents to anticipate the effects of their actions a…
3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning
Yuncong Yang, Han Yang, Jiachen Zhou +4
Constructing compact and informative 3D scene representations is essential for effective embodied exploration and reasoning, especially in complex environments over extended period…
FlexAttention for Efficient High-Resolution Vision-Language Models
Junyan Li, Delin Chen, Tianle Cai +5
Current high-resolution vision-language models encode images as high-resolution image tokens and exhaustively take all these tokens to compute attention, which significantly increa…
CoNav: A Benchmark for Human-Centered Collaborative Navigation
Changhao Li, Xinyu Sun, Peihao Chen +6
Human-robot collaboration, in which the robot intelligently assists the human with the upcoming task, is an appealing objective. To achieve this goal, the agent needs to be equippe…
MAGIC: Map-Guided Few-Shot Audio-Visual Acoustics Modeling
Diwei Huang, Kunyang Lin, Peihao Chen +2
Few-shot audio-visual acoustics modeling seeks to synthesize the room impulse response in arbitrary locations with few-shot observations. To sufficiently exploit the provided few-s…
3D-VLA: A 3D Vision-Language-Action Generative World Model
Haoyu Zhen, Xiaowen Qiu, Peihao Chen +5
Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by le…
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
Yining Hong, Zishuo Zheng, Peihao Chen +3
Human beings possess the capability to multiply a melange of multisensory cues while actively exploring and interacting with the 3D world. Current multi-modal large language models…
DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning
Kunyang Lin, Yufeng Wang, Peihao Chen +4
Learning optimal behavior policy for each agent in multi-agent systems is an essential yet difficult problem. Despite fruitful progress in multi-agent reinforcement learning, the c…
CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
Junyan Li, Delin Chen, Yining Hong +4
A remarkable ability of human beings resides in compositional reasoning, i.e., the capacity to make "infinite use of finite means". However, current large vision-language foundatio…
FGPrompt: Fine-grained Goal Prompting for Image-goal Navigation
Xinyu Sun, Peihao Chen, Jugang Fan +3
Learning to navigate to an image-specified goal is an important but challenging task for autonomous systems. The agent is required to reason the goal location from where a picture…
Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models
Peihao Chen, Xinyu Sun, Hongyan Zhi +5
We study the task of zero-shot vision-and-language navigation (ZS-VLN), a practical yet challenging problem in which an agent learns to navigate following a path described by langu…
Learning Vision-and-Language Navigation from YouTube Videos
Kunyang Lin, Peihao Chen, Diwei Huang +3
Vision-and-language navigation (VLN) requires an embodied agent to navigate in realistic 3D environments using natural language instructions. Existing VLN methods suffer from train…
Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition
Weidong Chen, Xiaofen Xing, Peihao Chen +1
This paper presents a paradigm that adapts general large-scale pretrained models (PTMs) to speech emotion recognition task. Although PTMs shed new light on artificial general intel…
Detecting the open-world objects with the help of the Brain
Shuailei Ma, Yuefeng Wang, Ying Wei +5
Open World Object Detection (OWOD) is a novel computer vision task with a considerable challenge, bridging the gap between classic object detection (OD) benchmarks and real-world o…
Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
Peihao Chen, Dongyu Ji, Kunyang Lin +4
We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions oft…
Learning Active Camera for Multi-Object Navigation
Peihao Chen, Dongyu Ji, Kunyang Lin +5
Getting robots to navigate to multiple objects autonomously is essential yet difficult in robot applications. One of the key challenges is how to explore environments efficiently w…
Masked Motion Encoding for Self-Supervised Video Representation Learning
Xinyu Sun, Peihao Chen, Liangwei Chen +4
How to learn discriminative video representation from unlabeled videos is challenging but crucial for video analysis. The latest attempts seek to learn a representation model by pr…
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning
Peihao Chen, Deng Huang, Dongliang He +5
We study unsupervised video representation learning that seeks to learn both motion and appearance features from unlabeled video only, which can be reused for downstream tasks such…
Location-aware Graph Convolutional Networks for Video Question Answering
Deng Huang, Peihao Chen, Runhao Zeng +3
We addressed the challenging task of video question answering, which requires machines to answer questions about videos in a natural language form. Previous state-of-the-art method…
Foley Music: Learning to Generate Music from Videos
Chuang Gan, Deng Huang, Peihao Chen +2
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key in…
Generating Visually Aligned Sound from Videos
Peihao Chen, Yang Zhang, Mingkui Tan +3
We focus on the task of generating sound from natural videos, and the sound should be both temporally and content-wise aligned with visual signals. This task is extremely challengi…
Dense Regression Network for Video Grounding
Runhao Zeng, Haoming Xu, Wenbing Huang +3
We address the problem of video grounding from natural language queries. The key challenge in this task is that one training video might only contain a few annotated starting/endin…
Self-supervised Moving Vehicle Tracking with Stereo Sound
Chuang Gan, Hang Zhao, Peihao Chen +2
Humans are able to localize objects in the environment using both visual and auditory cues, integrating information from multiple modalities into a common reference frame. We intro…