Low-Entropy Latent Variables Hurt Out-of-Distribution Performance

Nandi Schoots, Dylan Cope

We study the relationship between the entropy of intermediate representations and a model's robustness to distributional shift. We train models consisting of two feed-forward netwo…