4.4k citations · 7.7k across the 83 of their papers we have counts for
85 papers
GhostRNN: Reducing State Redundancy in RNN with Cheap Operations
Hang Zhou, Xiaoxu Zheng, Yunhe Wang +3
Recurrent neural network (RNNs) that are capable of modeling long-distance dependencies are widely used in various speech tasks, eg., keyword spotting (KWS) and speech enhancement…
GhostNetV3: Exploring the Training Strategies for Compact Models
Zhenhua Liu, Zhiwei Hao, Kai Han +2
Compact neural networks are specially designed for applications on edge devices with faster inference speed yet modest performance. However, training strategies of compact models a…
An Empirical Study of Scaling Law for OCR
Miao Rang, Zhenni Bi, Chuanjian Liu +2
The laws of model size, data volume, computation and model performance have been extensively studied in the field of Natural Language Processing (NLP). However, the scaling laws in…
One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation
Zhiwei Hao, Jianyuan Guo, Kai Han +4
Knowledge distillation~(KD) has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. However, most existing distillat…
Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism
Chengcheng Wang, Wei He, Ying Nie +4
In the past years, YOLO-series models have emerged as the leading approaches in the area of real-time object detection. Many studies pushed up the baseline to a higher level by mod…
Category Feature Transformer for Semantic Segmentation
Quan Tang, Chuanjian Liu, Fagui Liu +5
Aggregation of multi-stage features has been revealed to play a significant role in semantic segmentation. Unlike previous methods employing point-wise summation or concatenation f…
Less is More: Focus Attention for Efficient DETR
Dehua Zheng, Wenhui Dong, Hailin Hu +2
DETR-like models have significantly boosted the performance of detectors and even outperformed classical convolutional models. However, all tokens are treated equally without discr…
GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image
Mingjian Zhu, Hanting Chen, Qiangyu Yan +7
The extraordinary ability of generative models to generate photographic images has intensified concerns about the spread of disinformation, thereby leading to the demand for detect…
ParameterNet: Parameters Are All You Need
Kai Han, Yunhe Wang, Jianyuan Guo +1
The large-scale visual pretraining has significantly improve the performance of large vision models. However, we observe the \emph{low FLOPs pitfall} that the existing low-FLOPs mo…
VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale
Zhiwei Hao, Jianyuan Guo, Kai Han +3
The tremendous success of large models trained on extensive datasets demonstrates that scale is a key ingredient in achieving superior results. Therefore, the reflection on the rat…
VanillaNet: the Power of Minimalism in Deep Learning
Hanting Chen, Yunhe Wang, Jianyuan Guo +1
At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the…
Masked Image Modeling with Local Multi-Scale Reconstruction
Haoqing Wang, Yehui Tang, Yunhe Wang +3
Masked Image Modeling (MIM) achieves outstanding success in self-supervised representation learning. Unfortunately, MIM models typically have huge computational burden and slow lea…
BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons
Yixing Xu, Xinghao Chen, Yunhe Wang
This paper studies the problem of designing compact binary architectures for vision multi-layer perceptrons (MLPs). We provide extensive analysis on the difficulty of binarizing vi…
Redistribution of Weights and Activations for AdderNet Quantization
Ying Nie, Kai Han, Haikang Diao +3
Adder Neural Network (AdderNet) provides a new way for developing energy-efficient neural networks by replacing the expensive multiplications in convolution with cheaper additions…
FastMIM: Expediting Masked Image Modeling Pre-training for Vision
Jianyuan Guo, Kai Han, Han Wu +3
The combination of transformers and masked image modeling (MIM) pre-training framework has shown great potential in various vision tasks. However, the pre-training computational bu…
GhostNetV2: Enhance Cheap Operation with Long-Range Attention
Yehui Tang, Kai Han, Jianyuan Guo +3
Light-weight convolutional neural networks (CNNs) are specially designed for applications on mobile devices with faster inference speed. The convolutional operation can only captur…
AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets
Zhijun Tu, Xinghao Chen, Pengju Ren +1
This paper studies the Binary Neural Networks (BNNs) in which weights and activations are both binarized into 1-bit values, thus greatly reducing the memory usage and computational…
Vision GNN: An Image is Worth Graph of Nodes
Kai Han, Yunhe Wang, Jianyuan Guo +2
Network architecture plays a key role in the deep learning-based computer vision system. The widely-used convolutional neural network and transformer treat the image as a grid or s…
Source-Free Domain Adaptation via Distribution Estimation
Ning Ding, Yixing Xu, Yehui Tang +3
Domain Adaptation aims to transfer the knowledge learned from a labeled source domain to an unlabeled target domain whose data distributions are different. However, the training da…
Multimodal Token Fusion for Vision Transformers
Yikai Wang, Xinghao Chen, Lele Cao +3
Many adaptations of transformers have emerged to address the single-modal vision tasks, where self-attention modules are stacked to handle input sources like images. Intuitively, f…
Brain-inspired Multilayer Perceptron with Spiking Neurons
Wenshuo Li, Hanting Chen, Jianyuan Guo +2
Recently, Multilayer Perceptron (MLP) becomes the hotspot in the field of computer vision tasks. Without inductive bias, MLPs perform well on feature extraction and achieve amazing…
MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation
Min Zhong, Xinghao Chen, Xiaokang Chen +2
This paper studies the 3D instance segmentation problem, which has a variety of real-world applications such as robotics and augmented reality. Since the surroundings of 3D objects…
DropNAS: Grouped Operation Dropout for Differentiable Architecture Search
Weijun Hong, Guilin Li, Weinan Zhang +4
Neural architecture search (NAS) has shown encouraging results in automating the architecture design. Recently, DARTS relaxes the search process with a differentiable formulation t…
GhostNets on Heterogeneous Devices via Cheap Operations
Kai Han, Yunhe Wang, Chang Xu +4
Deploying convolutional neural networks (CNNs) on mobile devices is difficult due to the limited memory and computation resources. We aim to design efficient neural networks for he…