The impact of memory on learning sequence-to-sequence tasks

Abstract

The recent success of neural networks in natural language processing has drawn renewed attention to learning sequence-to-sequence (seq2seq) tasks. While there exists a rich literature that studies classification and regression tasks using solvable models of neural networks, seq2seq tasks have not yet been studied from this perspective. Here, we propose a simple model for a seq2seq task that has the advantage of providing explicit control over the degree of memory, or non-Markovianity, in the sequences -- the stochastic switching-Ornstein-Uhlenbeck (SSOU) model. We introduce a measure of non-Markovianity to quantify the amount of memory in the sequences. For a minimal auto-regressive (AR) learning model trained on this task, we identify two learning regimes corresponding to distinct phases in the stationary state of the SSOU process. These phases emerge from the interplay between two different time scales that govern the sequence statistics. Moreover, we observe that while increasing the integration window of the AR model always improves performance, albeit with diminishing returns, increasing the non-Markovianity of the input sequences can improve or degrade its performance. Finally, we perform experiments with recurrent and convolutional neural networks that show that our observations carry over to more complicated neural network architectures.

Code to reproduce our experiments available at https://github.com/alirezaseif/nonmarkovian_learning

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Estimating entropy production from waiting time distributions
• The Gaussian equivalence of generative models for learning with shallow neural networks
• The interplay between randomness and structure during learning in RNNs
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
• Modelling Active Non-Markovian Oscillations
• A Toolbox for Quantifying Memory in Dynamics Along Reaction Coordinates
• Probing transfer learning with a model of synthetic correlated datasets
• Asymptotics of Ridge (less) Regression under General Source Condition
• Quantifying non-Markovianity via conditional mutual information
• On the interplay between data structure and loss function in classification problems

Cited by in corpus (1)

• Mapping of attention mechanisms to a generalized Potts model