Memorization in Deep Neural Networks: Does the Loss Function matter?

Deep Patel, P. S. Sastry

Deep Neural Networks, often owing to the overparameterization, are shown to be capable of exactly memorizing even randomly labelled data. Empirical studies have also shown that non…