Anti-Distillation: Improving reproducibility of deep networks

Abstract

Deep networks have been revolutionary in improving performance of machine learning and artificial intelligence systems. Their high prediction accuracy, however, comes at a price of \emph{model irreproducibility\/} in very high levels that do not occur with classical linear models. Two models, even if they are supposedly identical, with identical architecture and identical trained parameter sets, and that are trained on the same set of training examples, while possibly providing identical average prediction accuracies, may predict very differently on individual, previously unseen, examples. \emph{Prediction differences\/} may be as large as the order of magnitude of the predictions themselves. Ensembles have been shown to somewhat mitigate this behavior, but without an extra push, may not be utilizing their full potential. In this work, a novel approach, \emph{Anti-Distillation\/}, is proposed to address irreproducibility in deep networks, where ensemble models are used to generate predictions. Anti-Distillation forces ensemble components away from one another by techniques like de-correlating their outputs over mini-batches of examples, forcing them to become even more different and more diverse. Doing so enhances the benefit of ensembles, making the final predictions more reproducible. Empirical results demonstrate substantial prediction difference reductions achieved by Anti-Distillation on benchmark and real datasets.

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• Knowledge Distillation: A Survey
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Understanding deep learning requires rethinking generalization
• Knowledge Distillation by On-the-Fly Native Ensemble
• Measuring the Effects of Data Parallelism on Neural Network Training
• Large scale distributed neural network training through online distillation
• Moonshine: Distilling with Cheap Convolutions
• Understanding and Improving Knowledge Distillation
• Critical Learning Periods in Deep Neural Networks
• Deterministic Implementations for Reproducibility in Deep Reinforcement Learning
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems

Cited by in corpus (5)

• Similarity of Neural Network Models: A Survey of Functional and Representational Measures
• Randomness In Neural Network Training: Characterizing The Impact of Tooling
• Smooth activations and reproducibility in deep networks
• Synthesizing Irreproducibility in Deep Networks
• Handling many conversions per click in modeling delayed feedback