5 papers
NormDirection: Restoring the Missing Query Norm in Vision Linear Attention
Weikang Meng, Yadan Luo, Liangyu Huo +4
Linear attention mitigates the quadratic complexity of softmax attention but suffers from a critical loss of expressiveness. We identify two primary causes: (1) The normalization o…
MirrorLA: Reflecting Feature Map for Vision Linear Attention
Weikang Meng, Liangyu Huo, Yadan Luo +3
Linear attention significantly reduces the computational complexity of Transformers from quadratic to linear, yet it consistently lags behind softmax-based attention in performance…
STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
Weikang Meng, Liangyu Huo, Yadan Luo +4
Linearizing pretrained large language models (LLMs) primarily relies on intra-layer hybrid attention mechanisms to alleviate the quadratic complexity of standard softmax attention.…
PolaFormer: Polarity-aware Linear Attention for Vision Transformers
Weikang Meng, Yadan Luo, Xin Li +2
Linear attention has emerged as a promising alternative to softmax-based attention, leveraging kernelized feature maps to reduce complexity from quadratic to linear in sequence len…
SC3D: Label-Efficient Outdoor 3D Object Detection via Single Click Annotation
Qiming Xia, Hongwei Lin, Wei Ye +4
LiDAR-based outdoor 3D object detection has received widespread attention. However, training 3D detectors from the LiDAR point cloud typically relies on expensive bounding box anno…