Dynamics and Reachability of Learning Tasks

Abstract

We compute the transition probability between two learning tasks, and show that it decomposes into two factors. The first depends on the geometry of the loss landscape of a model trained on each task, independent of any particular model used. This is related to an information theoretic distance function, but is insufficient to predict success in transfer learning, as nearby tasks can be unreachable via fine-tuning. The second factor depends on the ease of traversing the path between two tasks. With this dynamic component, we derive strict lower bounds on the complexity necessary to learn a task starting from the solution to another, which is one of the most common forms of transfer learning.

References in corpus (2)

• New insights and perspectives on the natural gradient method
• A PAC-Bayesian Tutorial with A Dropout Bound

Cited by in corpus (5)

• Geometric Dataset Distances via Optimal Transport
• Where is the Information in a Deep Neural Network?
• Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective
• Task-similarity Aware Meta-learning through Nonparametric Kernel Regression
• An Information-Geometric Distance on the Space of Tasks