Natural Statistics of Network Activations and Implications for Knowledge Distillation

Michael Rotman, Lior Wolf

In a matter that is analog to the study of natural image statistics, we study the natural statistics of the deep neural network activations at various layers. As we show, these sta…