22 citations · 22 across the 1 of their papers we have counts for
3 papers
cs.CL2020
Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation
Mitchell A. Gordon, Kevin Duh
We explore best practices for training small, memory efficient machine translation models with sequence-level knowledge distillation in the domain adaptation setting. While both do…
cs.CL2020
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Mitchell A. Gordon, Kevin Duh, Nicholas Andrews
Pre-trained universal feature extractors, such as BERT for natural language processing and VGG for computer vision, have become effective methods for improving deep learning models…
cs.CL2019★ 22 cited
Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation
Mitchell A. Gordon, Kevin Duh
Sequence-level knowledge distillation (SLKD) is a model compression technique that leverages large, accurate teacher models to train smaller, under-parameterized student models. Wh…