56 citations · 139 across the 13 of their papers we have counts for
6 papers · 1 filter
On-Policy Delta Distillation
Byeongho Heo, Jaehui Hwang, Sangdoo Yun +1
On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervisio…
DNNs May Determine Major Properties of Their Outputs Early, with Timing Possibly Driven by Bias
Song Park, Sanghyuk Chun, Byeongho Heo +1
This paper argues that deep neural networks (DNNs) mostly determine their outputs during the early stages of inference, where biases inherent in the model play a crucial role in sh…
Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching
Mingi Ji, Byeongho Heo, Sungrae Park
Knowledge distillation extracts general knowledge from a pre-trained teacher network and provides guidance to a target student network. Most studies manually tie intermediate featu…
AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
Byeongho Heo, Sanghyuk Chun, Seong Joon Oh +5
Normalization techniques are a boon for modern deep learning. They let weights converge more quickly with often better generalization performances. It has been argued that the norm…
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
Byeongho Heo, Minsik Lee, Sangdoo Yun +1
An activation boundary for a neuron refers to a separating hyperplane that determines whether the neuron is activated or deactivated. It has been long considered in neural networks…
Knowledge Distillation with Adversarial Samples Supporting Decision Boundary
Byeongho Heo, Minsik Lee, Sangdoo Yun +1
Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good te…