26 citations · 48 across the 10 of their papers we have counts for
11 papers
Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergence
Zhaowei Chen, Borui Zhao, Yuchen Ge +3
Online Knowledge Distillation (OKD) methods streamline the distillation training process into a single stage, eliminating the need for knowledge transfer from a pretrained teacher…
Cumulative Spatial Knowledge Distillation for Vision Transformers
Borui Zhao, Renjie Song, Jiajun Liang
Distilling knowledge from convolutional neural networks (CNNs) is a double-edged sword for vision transformers (ViTs). It boosts the performance since the image-friendly local-indu…
DOT: A Distillation-Oriented Trainer
Borui Zhao, Quan Cui, Renjie Song +1
Knowledge distillation transfers knowledge from a large model to a small one via task and distillation losses. In this paper, we observe a trade-off between task and distillation l…
Boosting Semi-Supervised Learning by Exploiting All Unlabeled Data
Yuhao Chen, Xin Tan, Borui Zhao +4
Semi-supervised learning (SSL) has attracted enormous attention due to its vast potential of mitigating the dependence on large labeled datasets. The latest methods (e.g., FixMatch…
Curriculum Temperature for Knowledge Distillation
Zheng Li, Xiang Li, Lingfeng Yang +5
Most existing distillation methods ignore the flexible role of the temperature in the loss function and fix it as a hyper-parameter that can be decided by an inefficient grid searc…
Efficient One Pass Self-distillation with Zipf's Label Smoothing
Jiajun Liang, Linze Li, Zhaodong Bing +4
Self-distillation exploits non-uniform soft supervision from itself during training and improves performance without any runtime cost. However, the overhead during training is ofte…
RecursiveMix: Mixed Learning with History
Lingfeng Yang, Xiang Li, Borui Zhao +2
Mix-based augmentation has been proven fundamental to the generalization of deep vision models. However, current augmentations only mix samples at the current data batch during tra…
Dynamic MLP for Fine-Grained Image Classification by Leveraging Geographical and Temporal Information
Lingfeng Yang, Xiang Li, Renjie Song +5
Fine-grained image classification is a challenging computer vision task where various species share similar visual appearances, resulting in misclassification if merely based on vi…
Decoupled Knowledge Distillation
Borui Zhao, Quan Cui, Renjie Song +2
State-of-the-art distillation methods are mainly based on distilling deep features from intermediate layers, while the significance of logit distillation is greatly overlooked. To…
Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective
Quan Cui, Bingchen Zhao, Zhao-Min Chen +5
This work simultaneously considers the discriminability and transferability properties of deep representations in the typical supervised learning task, i.e., image classification.…
Hierarchical Context Embedding for Region-based Object Detection
Zhao-Min Chen, Xin Jin, Borui Zhao +2
State-of-the-art two-stage object detectors apply a classifier to a sparse set of object proposals, relying on region-wise features extracted by RoIPool or RoIAlign as inputs. The…