25 citations · 50 across the 21 of their papers we have counts for
23 papers
Convolution for Large Language Models
Yuchuan Tian, Yingte Shu, Wei He +7
Large language models (LLMs) largely rely on Transformers, where self-attention provides global token interaction but does not explicitly encode the locality of natural language. W…
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
Yaojie Zhang, Jianuo Huang, Junlong Ke +5
Speculative decoding accelerates memory-bound LLM inference without quality degradation by using a fast drafter to propose multiple candidate tokens and the target model to verify…
Elastic-dLLM: Position Preserving Context Compression and Augmentation of Diffusion LLMs
Junyi Wu, Tianchen Zhao, Shaoqiu Zhang +3
Unlike autoregressive models, which generate one token at a time, dLLMs denoise a chunk of [MASK] tokens jointly and sample one or more tokens per step; despite enabling parallel d…
StreamingVLA: Streaming Vision-Language-Action Model with Action Flow Matching and Adaptive Early Observation
Yiran Shi, Dongqi Guo, Tianchen Zhao +8
Vision-language-action (VLA) models have demonstrated exceptional performance in natural language-driven perception and control. However, the high computational cost of VLA models…
SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer
Tongcheng Fang, Hanling Zhang, Ruiqi Xie +8
Diffusion Transformers have demonstrated remarkable performance in video generation. However, their long input sequences incur substantial latency due to the quadratic complexity o…
db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelism
Siqi Chen, Ke Hong, Tianchen Zhao +4
Scaling Diffusion Transformer (DiT) inference via sequence parallelism is critical for reducing latency in visual generation, but is severely hampered by workload imbalance when ap…
LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
Lianwei Yang, Haokun Lin, Tianchen Zhao +6
Diffusion Transformers (DiTs) have achieved impressive performance in text-to-image and text-to-video generation. However, their high computational cost and large parameter sizes p…
PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models
Tianchen Zhao, Ke Hong, Xinhao Yang +8
In visual generation, the quadratic complexity of attention mechanisms results in high memory and computational costs, especially for longer token sequences required in high-resolu…
PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
Tengxuan Liu, Shiyao Li, Jiayi Yang +7
Recently, significant progress has been made in developing reasoning-capable Large Language Models (LLMs) through long Chain-of-Thought (CoT) techniques. However, this long-CoT rea…
LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization
Rui Xie, Tianchen Zhao, Zhihang Yuan +5
Visual Autoregressive (VAR) has emerged as a promising approach in image generation, offering competitive potential and performance comparable to diffusion-based models. However, c…
DiTFastAttn: Attention Compression for Diffusion Transformer Models
Zhihang Yuan, Hanling Zhang, Pu Lu +6
Diffusion Transformers (DiT) excel at image and video generation but face computational challenges due to the quadratic complexity of self-attention operators. We propose DiTFastAt…
ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation
Tianchen Zhao, Tongcheng Fang, Haofeng Huang +10
Diffusion transformers have demonstrated remarkable performance in visual generation tasks, such as generating realistic images or videos based on textual instructions. However, la…
MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization
Tianchen Zhao, Xuefei Ning, Tongcheng Fang +6
Diffusion models have achieved significant visual generation quality. However, their significant computational and memory costs pose challenge for their application on resource-con…
FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models
Lin Zhao, Tianchen Zhao, Zinan Lin +4
In recent years, there has been significant progress in the development of text-to-image generative models. Evaluating the quality of the generative models is one essential step in…
Ada3D : Exploiting the Spatial Redundancy with Adaptive Inference for Efficient 3D Object Detection
Tianchen Zhao, Xuefei Ning, Ke Hong +8
Voxel-based methods have achieved state-of-the-art performance for 3D object detection in autonomous driving. However, their significant computational and memory costs pose a chall…
Dynamic Ensemble of Low-fidelity Experts: Mitigating NAS "Cold-Start"
Junbo Zhao, Xuefei Ning, Enshu Liu +7
Predictor-based Neural Architecture Search (NAS) employs an architecture performance predictor to improve the sample efficiency. However, predictor-based NAS suffers from the sever…
CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric Guidance
Tianchen Zhao, Niansong Zhang, Xuefei Ning +3
Transformers have gained much attention by outperforming convolutional neural networks in many 2D vision tasks. However, they are known to have generalization problems and rely on…
Discovering Robust Convolutional Architecture at Targeted Capacity: A Multi-Shot Approach
Xuefei Ning, Junbo Zhao, Wenshuo Li +4
Convolutional neural networks (CNNs) are vulnerable to adversarial examples, and studies show that increasing the model capacity of an architecture topology (e.g., width expansion)…
aw_nas: A Modularized and Extensible NAS framework
Xuefei Ning, Changcheng Tang, Wenshuo Li +7
Neural Architecture Search (NAS) has received extensive attention due to its capability to discover neural network architectures in an automated manner. aw_nas is an open-source Py…
BARS: Joint Search of Cell Topology and Layout for Accurate and Efficient Binary ARchitectures
Tianchen Zhao, Xuefei Ning, Xiangsheng Shi +6
Binary Neural Networks (BNNs) have received significant attention due to their promising efficiency. Currently, most BNN studies directly adopt widely-used CNN architectures, which…
Exploring the Potential of Low-bit Training of Convolutional Neural Networks
Kai Zhong, Xuefei Ning, Guohao Dai +5
In this work, we propose a low-bit training framework for convolutional neural networks, which is built around a novel multi-level scaling (MLS) tensor format. Our framework focuse…
DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation
Xuefei Ning, Tianchen Zhao, Wenshuo Li +3
Budgeted pruning is the problem of pruning under resource constraints. In budgeted pruning, how to distribute the resources across layers (i.e., sparsity allocation) is the key pro…
A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
Xuefei Ning, Yin Zheng, Tianchen Zhao +2
This work proposes a novel Graph-based neural ArchiTecture Encoding Scheme, a.k.a. GATES, to improve the predictor-based neural architecture search. Specifically, different from ex…