5 papers
The Surprising Effectiveness of Canonical Knowledge Distillation for Semantic Segmentation
Muhammad Ali, Kevin Alexander Laube, Madan Ravi Ganesh +3
Recent knowledge distillation (KD) methods for semantic segmentation introduce increasingly complex hand-crafted objectives, yet are typically evaluated under fixed iteration sched…
Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentation
Niclas Popp, Kevin Alexander Laube, Matthias Hein +1
Large foundation models trained on extensive datasets demonstrate strong zero-shot capabilities in various domains. Knowledge distillation has become an established tool for transf…
Attention Is All You Need For Mixture-of-Depths Routing
Advait Gadhikar, Souptik Kumar Majumdar, Niclas Popp +3
Advancements in deep learning are driven by training models with increasingly larger numbers of parameters, which in turn heightens the computational demands. To address this issue…
Single-Pass Object-Focused Data Selection
Niclas Popp, Dan Zhang, Jan Hendrik Metzen +2
While unlabeled image data is often plentiful, the costs of high-quality labels pose an important practical challenge: Which images should one select for labeling to use the annota…
Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data
Niclas Popp, Jan Hendrik Metzen, Matthias Hein
Multi-modal foundation models such as CLIP have showcased impressive zero-shot capabilities. However, their applicability in resource-constrained environments is limited due to the…