Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout

Zhao Chen, Jiquan Ngiam, Yanping Huang +4

The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights. However, these…