Publications (67)
DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving
Yingyan Li, Shuyao Shang, Weisong Liu +10
Scaling Vision-Language-Action (VLA) models on large-scale data offers a promising path to achieving a more generalized driving intelligence. However, VLA models are limited by a `…
Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding
Haoli Bai, Zhiguang Liu, Xiaojun Meng +9
Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding~(VDU). While various vision-language pre-tr…
Internet of Things Cloud: Architecture and Implementation
Lu Hou, Shaohang Zhao, Xiong Xiong +4
The Internet of Things (IoT), which enables common objects to be intelligent and interactive, is considered the next evolution of the Internet. Its pervasiveness and abilities to c…
FlatQuant: Flatness Matters for LLM Quantization
Yuxuan Sun, Ruikang Liu, Haoli Bai +10
Recently, quantization has been widely used for the compression and acceleration of large language models (LLMs). Due to the outliers in LLMs, it is crucial to flatten weights and…
FILIP: Fine-grained Interactive Language-Image Pre-Training
Lewei Yao, Runhui Huang, Lu Hou +7
Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks. Existing methods often model the cross-modal interaction either via…
Soft-Defined Heterogeneous Vehicular Network: Architecture and Challenges
Kan Zheng, Lu Hou, Hanlin Meng +3
Heterogeneous Vehicular NETworks (HetVNETs) can meet various quality-of-service (QoS) requirements for intelligent transport system (ITS) services by integrating different access n…
Embedding Compression in Recommender Systems: A Survey
Shiwei Li, Huifeng Guo, Xing Tang +4
To alleviate the problem of information explosion, recommender systems are widely deployed to provide personalized information filtering services. Usually, embedding tables are emp…
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding
Shuhuai Ren, Sishuo Chen, Shicheng Li +2
Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remai…
Towards Efficient Post-training Quantization of Pre-trained Language Models
Haoli Bai, Lu Hou, Lifeng Shang +3
Network quantization has gained increasing attention with the rapid growth of large pre-trained language models~(PLMs). However, most existing quantization methods for PLMs follow…
E-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
Tao Yuan, Haoli Bai, Yinfei Pan +5
With the increasing size of large language models, layer pruning has gained increased attention as a hardware-friendly approach for model compression. However, existing layer pruni…
Extending Context Window of Large Language Models via Semantic Compression
Weizhi Fei, Xueyan Niu, Pingyi Zhou +4
Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses. This constraint r…
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Wenliang Dai, Lu Hou, Lifeng Shang +3
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on v…
Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
Ruikang Liu, Yuxuan Sun, Manyi Zhang +5
Recent advancements in reasoning language models have demonstrated remarkable performance in complex tasks, but their extended chain-of-thought reasoning process increases inferenc…
DynaBERT: Dynamic BERT with Adaptive Width and Depth
Lu Hou, Zhiqi Huang, Lifeng Shang +3
The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive. To alleviate this problem, one appr…
The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs
Jierun Chen, Tiezheng Yu, Haoli Bai +11
Large vision-language models (VLMs) increasingly adopt post-training techniques such as long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL) to…
Design and Prototype Implementation of a Blockchain-Enabled LoRa System With Edge Computing
Lu Hou, Kan Zheng, Zhiming Liu +1
Efficiency and security have become critical issues during the development of the long-range (LoRa) system for Internet-of-Things (IoT) applications. The centralized work method in…
What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study
Keyu Lv, Manyi Zhang, Xiaobo Xia +6
Reasoning models excel at complex tasks such as coding and mathematics, yet their inference is often slow and token-inefficient. To improve the inference efficiency, post-training…
BinaryBERT: Pushing the Limit of BERT Quantization
Haoli Bai, Wei Zhang, Lu Hou +6
The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In thi…
DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
Zhe Liu, Runhui Huang, Rui Yang +6
Although multi-modal large language models (MLLMs) have shown strong capabilities across diverse domains, their application in generating fine-grained 3D perception and prediction…
DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Driving
Shuyao Shang, Bing Zhan, Yunfei Yan +9
We propose DynVLA, a driving VLA model that introduces a new CoT paradigm termed Dynamics CoT. DynVLA forecasts compact world dynamics before action generation, enabling more infor…
FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs
Haoran Lin, Xianzhi Yu, Kang Zhao +17
FlashAttention series has been widely applied in the inference of large language models (LLMs). However, FlashAttention series only supports the high-level GPU architectures, e.g.,…
InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search
Kaican Li, Lewei Yao, Jiannan Wu +7
The ability for AI agents to "think with images" requires a sophisticated blend of reasoning and perception. However, current open multimodal agents still largely fall short on the…
Loss-aware Weight Quantization of Deep Networks
Lu Hou, James T. Kwok
The huge size of deep networks hinders their use in small computing devices. In this paper, we consider compressing the network by weight quantization. We extend a recently propose…
UNIT: Unifying Image and Text Recognition in One Vision Encoder
Yi Zhu, Yanpeng Zhou, Chunwei Wang +4
Currently, vision encoder models like Vision Transformers (ViTs) typically excel at image recognition tasks but cannot simultaneously support text recognition like human visual rec…
An Intelligent Transaction Migration Scheme for RAFT-based Private Blockchain in Internet of Things Applications
Lu Hou, Xiaojun Xu, Kan Zheng +1
The integration of multi-access edge computing (MEC) and RAFT consensus makes it feasible to deploy blockchain on trustful base stations and gateways to provide efficient and tampe…
KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning
Hongling Xu, Qi Zhu, Heyuan Deng +6
Recent advances in large language model (LLM) post-training have leveraged two distinct paradigms to enhance reasoning capabilities: reinforcement learning (RL) and knowledge disti…
A Novel Rate and Channel Control Scheme Based on Data Extraction Rate for LoRa Networks
Qihao Zhou, Jinyu Xing, Lu Hou +2
Long Range (LoRa) has become one of the most popular Low Power Wide Area (LPWA) technologies, which provides a desirable trade-off among communication range, battery life, and depl…
TempCompass: Do Video LLMs Really Understand Videos?
Yuanxin Liu, Shicheng Li, Yi Liu +6
Recently, there is a surge in interest surrounding video large language models (Video LLMs). However, existing benchmarks fail to provide a comprehensive feedback on the temporal p…
Digital Retina for IoV Towards 6G: Architecture, Opportunities, and Challenges
Kan Zheng, Jie Mei, Haojun Yang +2
Vehicles are no longer isolated entities in traffic environments, thanks to the development of IoV powered by 5G networks and their evolution into 6G. However, it is not enough for…
Visually Guided Generative Text-Layout Pre-training for Document Intelligence
Zhiming Mao, Haoli Bai, Lu Hou +4
Prior study shows that pre-training techniques can boost the performance of visual document understanding (VDU), which typically requires models to gain abilities to perceive and r…
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling
Dongsheng Chen, Chaofan Tao, Lu Hou +3
Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks. However, the pre-training process is computationally expensive du…
MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric
Haokun Lin, Haoli Bai, Zhili Liu +5
Vision-language pre-trained models have achieved impressive performance on various downstream tasks. However, their large model sizes hinder their utilization on platforms with lim…
ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement
Runhui Huang, Chunwei Wang, Junwei Yang +8
We present ILLUME+ that leverages dual visual tokenization and a diffusion decoder to improve both deep semantic understanding and high-fidelity image generation. Existing unified…
Compression of Generative Pre-trained Language Models via Quantization
Chaofan Tao, Lu Hou, Wei Zhang +5
The increasing size of generative Pre-trained Language Models (PLMs) has greatly increased the demand for model compression. Despite various methods to compress BERT or its variant…
Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Models
Ruiying Peng, Xueyu Wu, Jing Lei +3
Multimodal large language models (MLLMs) often suffer from perceptual impairments under extended reasoning modes, particularly in visual question answering (VQA) tasks. We identify…
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Kai Chen, Yunhao Gou, Runhui Huang +28
GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language…
TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving
Wenqian Cui, Lei Zhu, Xiaohui Li +4
Full-Duplex Speech Language Models (FD-SLMs) are specialized foundation models designed to enable natural, real-time spoken interactions by modeling complex conversational turn-tak…
Design and Implementation of Open LoRa for IoT
Qihao Zhou, Kan Zheng, Lu Hou +2
Long Range (LoRa) network is emerging as one of the most promising Low Power Wide Area (LPWA) networks, since it enables the energy-constraint devices distributed over wide areas t…
DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learning
Wenxuan Shi, Haochen Tan, Chuqiao Kuang +7
Information seeking demands iterative evidence gathering and reflective reasoning, yet large language models (LLMs) still struggle with it in open-web question answering. Existing…
GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
Jingyi Wang, Lei Zhu, Tengjin Weng +8
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Language Models (LLMs) by leveraging direct outcome verification instead of l…
OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging
Yongxian Wei, Runxi Cheng, Weike Jin +7
Foundation models update slowly due to resource-intensive training, whereas domain-specific models evolve rapidly between releases. Model merging seeks to combine multiple expert m…
CTRL: Connect Collaborative and Language Model for CTR Prediction
Xiangyang Li, Bo Chen, Lu Hou +1
Traditional click-through rate (CTR) prediction models convert the tabular data into one-hot vectors and leverage the collaborative relations among features for inferring the user'…
OAC: Output-adaptive Calibration for Accurate Post-training Quantization
Ali Edalati, Alireza Ghaffari, Mahsa Ghazvini Nejad +4
Deployment of Large Language Models (LLMs) has major computational costs, due to their rapidly expanding size. Compression of LLMs reduces the memory footprint, latency, and energy…
Efficient Reasoning for Large Reasoning Language Models via Certainty-Guided Reflection Suppression
Jiameng Huang, Baijiong Lin, Guhao Feng +3
Recent Large Reasoning Language Models (LRLMs) employ long chain-of-thought reasoning with complex reflection behaviors, typically signaled by specific trigger words (e.g., "Wait"…
Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
Shiwei Li, Huifeng Guo, Lu Hou +5
Embedding tables are usually huge in click-through rate (CTR) prediction models. To train and deploy the CTR models efficiently and economically, it is necessary to compress their…
VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models
Shicheng Li, Lei Li, Shuhuai Ren +5
The ability to perceive how objects change over time is a crucial ingredient in human intelligence. However, current benchmarks cannot faithfully reflect the temporal understanding…
Faster and Better LLMs via Latency-Aware Test-Time Scaling
Zili Wang, Tianyu Zhang, Haoli Bai +5
Test-Time Scaling (TTS) has proven effective in improving the performance of Large Language Models (LLMs) during inference. However, existing research has overlooked the efficiency…
ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance
Chunwei Wang, Guansong Lu, Junwei Yang +5
In this paper, we introduce ILLUME, a unified multimodal large language model (MLLM) that seamlessly integrates multimodal understanding and generation capabilities within a single…
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact
Ruikang Liu, Haoli Bai, Haokun Lin +6
Large language models (LLMs) excel in natural language processing but demand intensive computation. To mitigate this, various quantization methods have been explored, yet they comp…
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Wei Zhang, Lu Hou, Yichun Yin +4
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.However, these models are both computation and memory…
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
Shuhuai Ren, Linli Yao, Shicheng Li +2
This work proposes TimeChat, a time-sensitive multimodal large language model specifically designed for long video understanding. Our model incorporates two key architectural contr…
Improved OOD Generalization via Adversarial Training and Pre-training
Mingyang Yi, Lu Hou, Jiacheng Sun +4
Recently, learning a model that generalizes well on out-of-distribution (OOD) data has attracted great attention in the machine learning community. In this paper, after defining OO…
Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models
Huanyu Wang, Jushi Kai, Haoli Bai +4
Vision-Language Models (VLMs) incur substantial computational overhead and inference latency due to the large number of vision tokens introduced by high-resolution image and video…
DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
Linli Yao, Lei Li, Shuhuai Ren +4
The visual projector, which bridges the vision and language modalities and facilitates cross-modal alignment, serves as a crucial component in MLLMs. However, measuring the effecti…
See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning
Yuxi Wei, Wei Huang, Qirui Chen +2
Spatial understanding is fundamental for embodied agents, yet most spatial VLMs and benchmarks remain offline-evaluating post-hoc QA over pre-recorded inputs and overlooking two cr…
DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding
Dong Zhuo, Wenzhao Zheng, Sicheng Zuo +4
With the growing adoption of vision-language-action models and world models in autonomous driving systems, scalable image tokenization becomes crucial as the interface for the visu…
Reliable and Efficient Autonomous Driving: the Need for Heterogeneous Vehicular Networks
Kan Zheng, Qiang Zheng, Haojun Yang +3
Autonomous driving technology has been regarded as a promising solution to reduce road accidents and traffic congestion, as well as to optimize the usage of fuel and lane. Reliable…
Loss-aware Binarization of Deep Networks
Lu Hou, Quanming Yao, James T. Kwok
Deep neural network models, though very powerful and highly successful, are computationally expensive in terms of space and time. Recently, there have been a number of attempts on…
FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
Yuanxin Liu, Lei Li, Shuhuai Ren +5
Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated vid…
Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
Jianhua Han, Meng Tian, Jiangtong Zhu +16
Autonomous driving heavily relies on accurate and robust spatial perception. Many failures arise from inaccuracies and instability, especially in long-tail scenarios and complex in…
An SDR-Based Test Platform for 5G NTN Prototyping and Validation
Lu Hou, Kan Zheng, Jie Mei +1
The integration of satellite communication into 5G has been formalized in 3GPP Release 17 through the specification of Non-Terrestrial Networks (NTN), marking a significant step to…
Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
Mingyang Yi, Lu Hou, Lifeng Shang +3
Data augmentation is an effective technique to improve the generalization of deep neural networks. However, previous data augmentation methods usually treat the augmented samples e…
Power Law in Sparsified Deep Neural Networks
Lu Hou, James T. Kwok
The power law has been observed in the degree distributions of many biological neural networks. Sparse deep neural networks, which learn an economical representation from the data,…
Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality
Sishuo Chen, Lei Li, Shuhuai Ren +5
Video paragraph captioning (VPC) involves generating detailed narratives for long videos, utilizing supportive modalities such as speech and event boundaries. However, the existing…
HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models
Runhui Huang, Xinpeng Ding, Chunwei Wang +7
High-resolution inputs enable Large Vision-Language Models (LVLMs) to discern finer visual details, enhancing their comprehension capabilities. To reduce the training and computati…
Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark
Jiaxi Gu, Xiaojun Meng, Guansong Lu +9
Vision-Language Pre-training (VLP) models have shown remarkable performance on various downstream tasks. Their success heavily relies on the scale of pre-trained cross-modal datase…
A Simple Linear Patch Revives Layer-Pruned Large Language Models
Xinrui Chen, Haoli Bai, Tao Yuan +7
Layer pruning has emerged as a widely used technique for compressing large language models (LLMs). However, existing layer pruning approaches often incur substantial performance de…