4.4k citations · 5.5k across the 6 of their papers we have counts for
6 papers
Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
Miao Rang, Zhenni Bi, Hang Zhou +6
Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, t…
Greedy Network Enlarging
Chuanjian Liu, Kai Han, An Xiao +4
Recent studies on deep convolutional neural networks present a simple paradigm of architecture design, i.e., models with more MACs typically achieve better accuracy, such as Effici…
Augmented Shortcuts for Vision Transformers
Yehui Tang, Kai Han, Chang Xu +4
Transformer models have achieved great progress on computer vision tasks recently. The rapid development of vision transformers is mainly contributed by their high representation a…
Transformer in Transformer
Kai Han, An Xiao, Enhua Wu +3
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the…
GhostSR: Learning Ghost Features for Efficient Image Super-Resolution
Ying Nie, Kai Han, Zhenhua Liu +2
Modern single image super-resolution (SISR) system based on convolutional neural networks (CNNs) achieves fancy performance while requires huge computational costs. The problem on…
A Survey on Visual Transformer
Kai Han, Yunhe Wang, Hanting Chen +10
Transformer, first applied to the field of natural language processing, is a type of deep neural network mainly based on the self-attention mechanism. Thanks to its strong represen…