Publications (49)
CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Driving
Enhui Ma, Lijun Zhou, Tao Tang +11
End-to-end planning methods are the de facto standard of the current autonomous driving system, while the robustness of the data-driven approaches suffers due to the notorious long…
How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
Kaicheng Yu, Rene Ranftl, Mathieu Salzmann
Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design…
DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies
Wei Song, Yuran Wang, Zijia Song +6
The differing representation spaces required for visual understanding and generation pose a challenge in unifying them within the autoregressive paradigm of large language models.…
OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving
Tao Tang, Enhui Ma, xia zhou +9
Autonomous driving has seen remarkable advancements, largely driven by extensive real-world data collection. However, acquiring diverse and corner-case data remains costly and inef…
OracleAnalyser: Analysing Implicit Semantics of Oracle Bone Scripts through MLLMs with Post-training
Zijia Song, Yelin Wang, Zhengyi Ma +5
With the advancement of artificial intelligence, research on oracle bone scripts has entered a new era. However, existing methods and benchmarks remain largely confined to recognit…
DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving
Enhui Ma, Jiahuan Zhang, Guantian Zheng +10
Multimodal Large Language Models (MLLMs) are rapidly becoming the intelligence brain of end-to-end autonomous driving systems. A key challenge is to assess whether MLLMs can truly…
Recurrent U-Net for Resource-Constrained Segmentation
Wei Wang, Kaicheng Yu, Joachim Hugonot +2
State-of-the-art segmentation methods rely on very deep networks that are not always easy to train without very large training datasets and tend to be relatively slow to run on sta…
Dyn-E: Local Appearance Editing of Dynamic Neural Radiance Fields
Shangzan Zhang, Sida Peng, Yinji ShenTu +5
Recently, the editing of neural radiance fields (NeRFs) has gained considerable attention, but most prior works focus on static scenes while research on the appearance editing of d…
SR-LLM: Rethinking the Structured Representation in Large Language Model
Jiahuan Zhang, Tianheng Wang, Hanqing Wu +7
Structured representations, exemplified by Abstract Meaning Representation (AMR), have long been pivotal in computational linguistics. However, their role remains ambiguous in the…
BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout
Kairui Yang, Enhui Ma, Jibin Peng +3
Using synthesized images to boost the performance of perception models is a long-standing research challenge in computer vision. It becomes more eminent in visual-centric autonomou…
Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation
Enhui Ma, Lijun Zhou, Tao Tang +9
Using generative models to synthesize new data has become a de-facto standard in autonomous driving to address the data scarcity issue. Though existing approaches are able to boost…
FusionFormer: A Multi-sensory Fusion in Bird's-Eye-View and Temporal Consistent Transformer for 3D Object Detection
Chunyong Hu, Hang Zheng, Kun Li +11
Multi-sensor modal fusion has demonstrated strong advantages in 3D object detection tasks. However, existing methods that fuse multi-modal features require transforming features in…
NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy
Yash Mehta, Colin White, Arber Zela +6
The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural archit…
THE-Tree: Can Tracing Historical Evolution Enhance Scientific Verification and Reasoning?
Xin Wang, Jiyao Liu, Yulong Xiao +5
Large Language Models (LLMs) are accelerating scientific idea generation, but rigorously evaluating these numerous, often superficial, AI-generated propositions for novelty and fac…
Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask
Shangzan Zhang, Sida Peng, Tianrun Chen +5
We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single ima…
Strategic priorities for transformative progress in advancing biology with proteomics and artificial intelligence
Yingying Sun, Jun A, Zhiwei Liu +59
Artificial intelligence (AI) is transforming scientific research, including proteomics. Advances in mass spectrometry (MS)-based proteomics data quality, diversity, and scale, comb…
Overcoming Multi-Model Forgetting
Yassine Benyahia, Kaicheng Yu, Kamil Bennani-Smires +4
We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performan…
Baichuan-Omni Technical Report
Yadong Li, Haoze Sun, Mingan Lin +23
The salient multimodal capabilities and interactive experience of GPT-4o highlight its critical role in practical applications, yet it lacks a high-performing open-source counterpa…
MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Driving
Haiguang Wang, Daqi Liu, Hongwei Xie +5
In recent years, data-driven techniques have greatly advanced autonomous driving systems, but the need for rare and diverse training data remains a challenge, requiring significant…
BEVHeight: A Robust Framework for Vision-based Roadside 3D Object Detection
Lei Yang, Kaicheng Yu, Tao Tang +5
While most recent autonomous driving system focuses on developing perception methods on ego-vehicle sensors, people tend to overlook an alternative approach to leverage intelligent…
BioKGBench: A Knowledge Graph Checking Benchmark of AI Agent for Biomedical Science
Xinna Lin, Siqi Ma, Junjie Shan +5
Pursuing artificial intelligence for biomedical science, a.k.a. AI Scientist, draws increasing attention, where one common approach is to build a copilot agent driven by Large Lang…
Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search
Kaicheng Yu, Rene Ranftl, Mathieu Salzmann
Weight sharing has become a de facto standard in neural architecture search because it enables the search to be done on commodity hardware. However, recent works have empirically s…
AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis
Tao Tang, Guangrun Wang, Yixing Lao +5
Neural implicit fields have been a de facto standard in novel view synthesis. Recently, there exist some methods exploring fusing multiple modalities within a single field, aiming…
An Analysis of Super-Net Heuristics in Weight-Sharing NAS
Kaicheng Yu, René Ranftl, Mathieu Salzmann
Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design…
CRUISE: Cooperative Reconstruction and Editing in V2X Scenarios using Gaussian Splatting
Haoran Xu, Saining Zhang, Peishuo Li +15
Vehicle-to-everything (V2X) communication plays a crucial role in autonomous driving, enabling cooperation between vehicles and infrastructure. While simulation has significantly c…
FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving
Tengju Ye, Wei Jing, Chunyong Hu +11
Building a multi-modality multi-task neural network toward accurate and robust performance is a de-facto standard in perception task of autonomous driving. However, leveraging such…
DiVE: DiT-based Video Generation with Enhanced Control
Junpeng Jiang, Gangyi Hong, Lijun Zhou +10
Generating high-fidelity, temporally consistent videos in autonomous driving scenarios faces a significant challenge, e.g. problematic maneuvers in corner cases. Despite recent vid…
Channel-wise Vector Quantization
Wei Song, Tianhang Wang, Yitong Chen +5
We present Channel-wise Vector Quantization (CVQ), a novel image tokenization paradigm that replaces patch-wise tokens with channel-wise tokens. Unlike conventional vector quantiza…
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework
Tingting Liang, Hongwei Xie, Kaicheng Yu +6
Fusing the camera and LiDAR information has become a de-facto standard for 3D object detection tasks. Current methods rely on point clouds from the LiDAR sensor as queries to lever…
Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection
Kaicheng Yu, Tang Tao, Hongwei Xie +10
There are two critical sensors for 3D perception in autonomous driving, the camera and the LiDAR. The camera provides rich semantic information such as color, texture, and the LiDA…
Statistically Motivated Second Order Pooling
Kaicheng Yu, Mathieu Salzmann
Second-order pooling, a.k.a.~bilinear pooling, has proven effective for deep learning based visual recognition. However, the resulting second-order networks yield a final represent…
CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
Yuxuan Zhu, Antony Kellermann, Dylan Bowman +13
Large language model (LLM) agents are increasingly capable of autonomously conducting cyberattacks, posing significant threats to existing applications. This growing risk highlight…
BEVHeight++: Toward Robust Visual Centric 3D Object Detection
Lei Yang, Tao Tang, Jun Li +6
While most recent autonomous driving system focuses on developing perception methods on ego-vehicle sensors, people tend to overlook an alternative approach to leverage intelligent…
LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields
Tang Tao, Longfei Gao, Guangrun Wang +7
We introduce a new task, novel view synthesis for LiDAR sensors. While traditional model-based LiDAR simulators with style-transfer neural networks can be applied to render novel v…
Evaluating the Search Phase of Neural Architecture Search
Kaicheng Yu, Christian Sciuto, Martin Jaggi +2
Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and val…
Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving
Mingyi Wang, Jingke Wang, Tengju Ye +2
Recent breakthroughs in large language models (LLMs) have not only advanced natural language processing but also inspired their application in domains with structurally similar pro…
DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling
Bicheng Guo, Shuxuan Guo, Miaojing Shi +4
Differentiable architecture search (DARTS) has been a mainstream direction in automatic machine learning. Since the discovery that original DARTS will inevitably converge to poor a…
DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing
Dianyi Wang, Ruihang Li, Feng Han +17
Current unified multimodal models for image generation and editing typically rely on massive parameter scales (e.g., >10B), entailing prohibitive training costs and deployment foot…
DriveMRP: Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Prediction
Zhiyi Hou, Enhui Ma, Fang Li +11
Autonomous driving has seen significant progress, driven by extensive real-world data. However, in long-tail scenarios, accurately predicting the safety of the ego vehicle's future…
Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better
Dianyi Wang, Wei Song, Yikun Wang +4
Typical large vision-language models (LVLMs) apply autoregressive supervision solely to textual sequences, without fully incorporating the visual modality into the learning process…
M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmark
Wei Song, Yadong Li, Jianhua Xu +8
As recent multi-modality large language models (MLLMs) have shown formidable proficiency on various complex tasks, there has been increasing attention on debating whether these mod…
OpenSight: A Simple Open-Vocabulary Framework for LiDAR-Based Object Detection
Hu Zhang, Jianhua Xu, Tao Tang +4
Traditional LiDAR-based object detection research primarily focuses on closed-set scenarios, which falls short in complex real-world applications. Directly transferring existing 2D…
Second-order Convolutional Neural Networks
Kaicheng Yu, Mathieu Salzmann
Convolutional Neural Networks (CNNs) have been successfully applied to many computer vision tasks, such as image classification. By performing linear combinations and element-wise…
Knowledge Distillation via the Target-aware Transformer
Sihao Lin, Hongwei Xie, Bing Wang +4
Knowledge distillation becomes a de facto standard to improve the performance of small neural networks. Most of the previous works propose to regress the representational features…
: Intelligent Agent for Automated Pharmaceutical Patent Analysis
Xin Wang, Yifan Zhang, Xiaojing Zhang +5
Pharmaceutical patents play a vital role in biochemical industries, especially in drug discovery, providing researchers with unique early access to data, experimental results, and…
Beyond One Glance: Gated Recurrent Architecture for Hand Segmentation
Wei Wang, Kaicheng Yu, Joachim Hugonot +2
As mixed reality is gaining increased momentum, the development of effective and efficient solutions to egocentric hand segmentation is becoming critical. Traditional segmentation…
Learning Self-Regularized Adversarial Views for Self-Supervised Vision Transformers
Tao Tang, Changlin Li, Guangrun Wang +3
Automatic data augmentation (AutoAugment) strategies are indispensable in supervised data-efficient training protocols of vision transformers, and have led to state-of-the-art resu…
Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models
Jiahuan Zhang, Shunwen Bai, Tianheng Wang +4
Humans naturally possess the spatial reasoning ability to form and manipulate images and structures of objects in space. There is an increasing effort to endow Vision-Language Mode…
Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training
Xiaoyang Wu, Zhuotao Tian, Xin Wen +4
The rapid advancement of deep learning models often attributes to their ability to leverage massive training data. In contrast, such privilege has not yet fully benefited 3D deep l…