Jitter: Random Jittering Loss Function

Abstract

Regularization plays a vital role in machine learning optimization. One novel regularization method called flooding makes the training loss fluctuate around the flooding level. It intends to make the model continue to random walk until it comes to a flat loss landscape to enhance generalization. However, the hyper-parameter flooding level of the flooding method fails to be selected properly and uniformly. We propose a novel method called Jitter to improve it. Jitter is essentially a kind of random loss function. Before training, we randomly sample the Jitter Point from a specific probability distribution. The flooding level should be replaced by Jitter point to obtain a new target function and train the model accordingly. As Jitter point acting as a random factor, we actually add some randomness to the loss function, which is consistent with the fact that there exists innumerable random behaviors in the learning process of the machine learning model and is supposed to make the model more robust. In addition, Jitter performs random walk randomly which divides the loss curve into small intervals and then flipping them over, ideally making the loss curve much flatter and enhancing generalization ability. Moreover, Jitter can be a domain-, task-, and model-independent regularization method and train the model effectively after the training error reduces to zero. Our experimental results show that Jitter method can improve model performance more significantly than the previous flooding method and make the test loss curve descend twice.

arXiv commentIJCNN 2021

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Reconciling modern machine learning practice and the bias-variance trade-off
• DropBlock: A regularization method for convolutional networks
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• Do We Need Zero Training Loss After Achieving Zero Training Error?
• Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
• Towards Understanding Label Smoothing
• Batch Group Normalization
• Data Augmentation for Object Detection via Differentiable Neural Rendering