160 citations · 425 across the 18 of their papers we have counts for
5 papers · 1 filter
Pruning Self-attentions into Convolutional Layers in Single Path
Haoyu He, Jianfei Cai, Jing Liu +4
Vision Transformers (ViTs) have achieved impressive performance over various computer vision tasks. However, modeling global correlations with multi-head self-attention (MSA) layer…
Mesa: A Memory-saving Training Framework for Transformers
Zizheng Pan, Peng Chen, Haoyu He +3
There has been an explosion of interest in designing high-performance Transformers. While Transformers have delivered significant performance improvements, training such networks i…
Less is More: Pay Less Attention in Vision Transformers
Zizheng Pan, Bohan Zhuang, Haoyu He +2
Transformers have become one of the dominant architectures in deep learning, particularly as a powerful alternative to convolutional neural networks (CNNs) in computer vision. Howe…
Scalable Vision Transformers with Hierarchical Pooling
Zizheng Pan, Bohan Zhuang, Jing Liu +2
The recently proposed Visual image Transformers (ViT) with pure attention have achieved promising performance on image recognition tasks, such as image classification. However, the…
Single-path Bit Sharing for Automatic Loss-aware Model Compression
Jing Liu, Bohan Zhuang, Peng Chen +3
Network pruning and quantization are proven to be effective ways for deep model compression. To obtain a highly compact model, most methods first perform network pruning and then c…