5 papers
Learning Spatial Decay for Vision Transformers
Yuxin Mao, Zhen Qin, Jinxing Zhou +4
Vision Transformers (ViTs) have revolutionized computer vision, yet their self-attention mechanism lacks explicit spatial inductive biases, leading to suboptimal performance on spa…
Elucidating the Design Space of Decay in Linear Attention
Zhen Qin, Xuyang Shen, Yiran Zhong
This paper presents a comprehensive investigation into the decay mechanisms inherent in linear complexity sequence models. We systematically delineate the design space of decay mec…
Autoregressive Image Generation with Linear Complexity: A Spatial-Aware Decay Perspective
Yuxin Mao, Zhen Qin, Jinxing Zhou +6
Autoregressive (AR) models have garnered significant attention in image generation for their ability to effectively capture both local and global structures within visual data. How…
Linear Attention Sequence Parallelism
Weigao Sun, Zhen Qin, Dong Li +3
Sequence parallelism (SP) serves as a prevalent strategy to handle long sequences that exceed the memory limit of a single device. However, for linear sequence modeling methods lik…
Deep Non-rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling
Hui Deng, Jiawei Shi, Zhen Qin +2
Non-Rigid Structure-from-Motion (NRSfM) is a classic 3D vision problem, where a 2D sequence is taken as input to estimate the corresponding 3D sequence. Recently, the deep neural n…