Publications (44)
GIA-Net: Global Information Aware Network for Low-light Imaging
Zibo Meng, Runsheng Xu, Chiu Man Ho
It is extremely challenging to acquire perceptually plausible images under low-light conditions due to low SNR. Most recently, U-Nets have shown promising results for low-light ima…
S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation
Yichen Xie, Runsheng Xu, Tong He +9
The latest advancements in multi-modal large language models (MLLMs) have spurred a strong renewed interest in end-to-end motion planning approaches for autonomous driving. Many en…
V2X-DGW: Domain Generalization for Multi-agent Perception under Adverse Weather Conditions
Baolu Li, Jinlong Li, Xinyu Liu +5
Current LiDAR-based Vehicle-to-Everything (V2X) multi-agent perception systems have shown the significant success on 3D object detection. While these models perform well in the tra…
Street-View Image Generation from a Bird's-Eye View Layout
Alexander Swerdlow, Runsheng Xu, Bolei Zhou
Bird's-Eye View (BEV) Perception has received increasing attention in recent years as it provides a concise and unified spatial representation across views and benefits a diverse s…
Model-Agnostic Multi-Agent Perception Framework
Runsheng Xu, Weizhe Chen, Hao Xiang +2
Existing multi-agent perception systems assume that every agent utilizes the same model with identical parameters and architecture. The performance can be degraded with different p…
FedBEVT: Federated Learning Bird's Eye View Perception Transformer in Road Traffic Systems
Rui Song, Runsheng Xu, Andreas Festag +2
Bird's eye view (BEV) perception is becoming increasingly important in the field of autonomous driving. It uses multi-view camera data to learn a transformer model that directly pr…
The OpenCDA Open-source Ecosystem for Cooperative Driving Automation Research
Runsheng Xu, Hao Xiang, Xu Han +4
Advances in Single-vehicle intelligence of automated driving have encountered significant challenges because of limited capabilities in perception and interaction with complex traf…
Automated Driving Systems Data Acquisition and Processing Platform
Xin Xia, Zonglin Meng, Xu Han +5
This paper presents an automated driving system (ADS) data acquisition and processing platform for vehicle trajectory extraction, reconstruction, and evaluation based on connected…
Hierarchical Road Topology Learning for Urban Map-less Driving
Li Zhang, Faezeh Tafazzoli, Gunther Krehl +4
The majority of current approaches in autonomous driving rely on High-Definition (HD) maps which detail the road geometry and surrounding area. Yet, this reliance is one of the obs…
Enhanced Motion Forecasting with Plug-and-Play Multimodal Large Language Models
Katie Luo, Jingwei Ji, Tong He +4
Current autonomous driving systems rely on specialized models for perceiving and predicting motion, which demonstrate reliable performance in standard conditions. However, generali…
Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perception
Si Liu, Chen Gao, Yuan Chen +8
Vehicle-to-everything (V2X) autonomous driving opens up a promising direction for developing a new generation of intelligent transportation systems. Collaborative perception (CP) a…
OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication
Runsheng Xu, Hao Xiang, Xin Xia +3
Employing Vehicle-to-Vehicle communication to enhance perception performance in self-driving technology has attracted considerable attention recently; however, the absence of a sui…
Breaking Data Silos: Cross-Domain Learning for Multi-Agent Perception from Independent Private Sources
Jinlong Li, Baolu Li, Xinyu Liu +3
The diverse agents in multi-agent perception systems may be from different companies. Each company might use the identical classic neural network architecture based encoder for fea…
ROMNet: Renovate the Old Memories
Runsheng Xu, Zhengzhong Tu, Yuanqi Du +5
Renovating the memories in old photos is an intriguing research topic in computer vision fields. These legacy images often suffer from severe and commingled degradations such as cr…
Lane Boundary Geometry Extraction from Satellite Imagery
Andi Zang, Runsheng Xu, Zichen Li +1
Autonomous driving car is becoming more of a reality, as a key component,high-definition(HD) maps shows its value in both market place and industry. Even though HD maps generation…
CoST: Efficient Collaborative Perception From Unified Spatiotemporal Perspective
Zongheng Tang, Yi Liu, Yifan Sun +4
Collaborative perception shares information among different agents and helps solving problems that individual agents may face, e.g., occlusions and small sensing range. Prior metho…
CoCMT: Communication-Efficient Cross-Modal Transformer for Collaborative Perception
Rujia Wang, Xiangbo Gao, Hao Xiang +2
Multi-agent collaborative perception enhances each agent perceptual capabilities by sharing sensing information to cooperatively perform robot perception tasks. This approach has p…
MAGNIFIED: RL Fine-tuning of Multimodal Large Language Models for Motion Planning
Letian Chen, Yiren Lu, Justin Fu +5
Multi-modal Large Language Models (MLLMs) have demonstrated remarkable capabilities in semantic understanding and common sense reasoning, making them promising candidates for solvi…
HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative perception with vision transformer
Hao Xiang, Runsheng Xu, Jiaqi Ma
Vehicle-to-Vehicle technologies have enabled autonomous vehicles to share information to see through occlusions, greatly enhancing perception performance. Nevertheless, existing wo…
Diffusion Models: A Comprehensive Survey of Methods and Applications
Ling Yang, Zhilong Zhang, Yang Song +6
Diffusion models have emerged as a powerful new family of deep generative models with record-breaking performance in many applications, including image synthesis, video generation,…
EMMA: End-to-End Multimodal Model for Autonomous Driving
Jyh-Jing Hwang, Runsheng Xu, Hubert Lin +11
We introduce EMMA, an End-to-end Multimodal Model for Autonomous driving. Built upon a multi-modal large language model foundation like Gemini, EMMA directly maps raw camera sensor…
Domain Adaptation based Object Detection for Autonomous Driving in Foggy and Rainy Weather
Jinlong Li, Runsheng Xu, Xinyu Liu +5
Typically, object detection methods for autonomous driving that rely on supervised learning make the assumption of a consistent feature distribution between the training and testin…
V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception
Runsheng Xu, Xin Xia, Jinlong Li +10
Modern perception systems of autonomous vehicles are known to be sensitive to occlusions and lack the capability of long perceiving range. It has been one of the key bottlenecks th…
NeckSense: A Multi-Sensor Necklace for Detecting Eating Activities in Free-Living Conditions
Shibo Zhang, Yuqi Zhao, Dzung Tri Nguyen +4
We present the design, implementation, and evaluation of a multi-sensor low-power necklace 'NeckSense' for automatically and unobtrusively capturing fine-grained information about…
V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer
Runsheng Xu, Hao Xiang, Zhengzhong Tu +3
In this paper, we investigate the application of Vehicle-to-Everything (V2X) communication to improve the perception performance of autonomous vehicles. We present a robust coopera…
CoMamba: Real-time Cooperative Perception Unlocked with State Space Models
Jinlong Li, Xinyu Liu, Baolu Li +4
Cooperative perception systems play a vital role in enhancing the safety and efficiency of vehicular autonomy. Although recent studies have highlighted the efficacy of vehicle-to-e…
Collaboration Helps Camera Overtake LiDAR in 3D Detection
Yue Hu, Yifan Lu, Runsheng Xu +3
Camera-only 3D detection provides an economical solution with a simple configuration for localizing objects in 3D space compared to LiDAR-based detection systems. However, a major…
OpenCDA:An Open Cooperative Driving Automation Framework Integrated with Co-Simulation
Runsheng Xu, Yi Guo, Xu Han +3
Although Cooperative Driving Automation (CDA) has attracted considerable attention in recent years, there remain numerous open challenges in this field. The gap between existing si…
S2R-ViT for Multi-Agent Cooperative Perception: Bridging the Gap from Simulation to Reality
Jinlong Li, Runsheng Xu, Xinyu Liu +4
Due to the lack of enough real multi-agent data and time-consuming of labeling, existing multi-agent cooperative perception algorithms usually select the simulated sensor data for…
Towards Better Driver Safety: Empowering Personal Navigation Technologies with Road Safety Awareness
Runsheng Xu, Shibo Zhang, Yue Zhao +4
Recent research has found that navigation systems usually assume that all roads are equally safe, directing drivers to dangerous routes, which led to catastrophic consequences. To…
STAMP: Scalable Task And Model-agnostic Collaborative Perception
Xiangbo Gao, Runsheng Xu, Jiachen Li +3
Perception is crucial for autonomous driving, but single-agent perception is often constrained by sensors' physical limitations, leading to degraded performance under severe occlus…
Domain Adaptive Object Detection for Autonomous Driving under Foggy Weather
Jinlong Li, Runsheng Xu, Jin Ma +3
Most object detection methods for autonomous driving usually assume a consistent feature distribution between training and testing data, which is not always the case when weathers…
V2X-Real: a Large-Scale Dataset for Vehicle-to-Everything Cooperative Perception
Hao Xiang, Zhaoliang Zheng, Xin Xia +15
Recent advancements in Vehicle-to-Everything (V2X) technologies have enabled autonomous vehicles to share sensing information to see through occlusions, greatly boosting the percep…
Analyzing Infrastructure LiDAR Placement with Realistic LiDAR Simulation Library
Xinyu Cai, Wentao Jiang, Runsheng Xu +4
Recently, Vehicle-to-Everything(V2X) cooperative perception has attracted increasing attention. Infrastructure sensors play a critical role in this research field; however, how to…
Learning for Vehicle-to-Vehicle Cooperative Perception under Lossy Communication
Jinlong Li, Runsheng Xu, Xinyu Liu +4
Deep learning has been widely used in the perception (e.g., 3D object detection) of intelligent vehicle driving. Due to the beneficial Vehicle-to-Vehicle (V2V) communication, the d…
Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving
Jinlong Li, Baolu Li, Zhengzhong Tu +5
Vision-centric perception systems for autonomous driving have gained considerable attention recently due to their cost-effectiveness and scalability, especially compared to LiDAR-b…
V2XP-ASG: Generating Adversarial Scenes for Vehicle-to-Everything Perception
Hao Xiang, Runsheng Xu, Xin Xia +3
Recent advancements in Vehicle-to-Everything communication technology have enabled autonomous vehicles to share sensory information to obtain better perception performance. With th…
Optimizing the Placement of Roadside LiDARs for Autonomous Driving
Wentao Jiang, Hao Xiang, Xinyu Cai +5
Multi-agent cooperative perception is an increasingly popular topic in the field of autonomous driving, where roadside LiDARs play an essential role. However, how to optimize the p…
DUSA: Decoupled Unsupervised Sim2Real Adaptation for Vehicle-to-Everything Collaborative Perception
Xianghao Kong, Wentao Jiang, Jinrang Jia +3
Vehicle-to-Everything (V2X) collaborative perception is crucial for autonomous driving. However, achieving high-precision V2X perception requires a significant amount of annotated…
Holistic Grid Fusion Based Stop Line Estimation
Runsheng Xu, Faezeh Tafazzoli, Li Zhang +3
Intersection scenarios provide the most complex traffic situations in Autonomous Driving and Driving Assistance Systems. Knowing where to stop in advance in an intersection is an e…
Bridging the Domain Gap for Multi-Agent Perception
Runsheng Xu, Jinlong Li, Xiaoyu Dong +2
Existing multi-agent perception algorithms usually select to share deep neural features extracted from raw sensing data between agents, achieving a trade-off between accuracy and c…
Pik-Fix: Restoring and Colorizing Old Photos
Runsheng Xu, Zhengzhong Tu, Yuanqi Du +6
Restoring and inpainting the visual memories that are present, but often impaired, in old photos remains an intriguing but unsolved research topic. Decades-old photos often suffer…
WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios
Runsheng Xu, Hubert Lin, Wonseok Jeon +11
Vision-based end-to-end (E2E) driving has garnered significant interest in the research community due to its scalability and synergy with multimodal large language models (MLLMs).…
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers
Runsheng Xu, Zhengzhong Tu, Hao Xiang +3
Bird's eye view (BEV) semantic segmentation plays a crucial role in spatial sensing for autonomous driving. Although recent literature has made significant progress on BEV map unde…