Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation

Wenhui Chen, Zhifeng Li, Jie Zhou +5

A compressed student has two shapes that need not agree: the weight it deploys at inference and the weight family its training can reach. We show that a state-of-the-art weight-inh…