Generalization in multitask deep neural classifiers: a statistical physics approach

Abstract

A proper understanding of the striking generalization abilities of deep neural networks presents an enduring puzzle. Recently, there has been a growing body of numerically-grounded theoretical work that has contributed important insights to the theory of learning in deep neural nets. There has also been a recent interest in extending these analyses to understanding how multitask learning can further improve the generalization capacity of deep neural nets. These studies deal almost exclusively with regression tasks which are amenable to existing analytical techniques. We develop an analytic theory of the nonlinear dynamics of generalization of deep neural networks trained to solve classification tasks using softmax outputs and cross-entropy loss, addressing both single task and multitask settings. We do so by adapting techniques from the statistical physics of disordered systems, accounting for both finite size datasets and correlated outputs induced by the training dynamics. We discuss the validity of our theoretical results in comparison to a comprehensive suite of numerical experiments. Our analysis provides theoretical support for the intuition that the performance of multitask learning is determined by the noisiness of the tasks and how well their input features align with each other. Highly related, clean tasks benefit each other, whereas unrelated, clean tasks can be detrimental to individual task performance.

Accepted to NeurIPS 2019

References in corpus (8)

• Understanding Deep Convolutional Networks
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Multi-Task Learning as Multi-Objective Optimization
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• Adapting Auxiliary Losses Using Gradient Similarity
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Generalisation dynamics of online learning in over-parameterised neural networks

Cited by in corpus (4)

• Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs
• Machine learning for complete intersection Calabi-Yau manifolds: a methodological study
• A Close Look at Deep Learning with Small Data
• Multitask Learning Strengthens Adversarial Robustness