When Losses Align: Gradient-Based Composite Loss Weighting for Efficient Pretraining

Ivan Karpukhin, Andrey Savchenko

Modern deep models are often pretrained on large-scale data with missing labels using composite objectives, where the relative weights of multiple loss terms act as hyperparameters…