The Implicit Bias of Logit Regularization

Alon Beck, Yohai Bar Sinai, Noam Levi

Logit regularization, the addition of a convex penalty directly in logit space, is widely used in modern classifiers, with label smoothing as a prominent example. While such method…