The impact of memory on learning sequence-to-sequence tasks

Abstract

The recent success of neural networks in natural language processing has drawn renewed attention to learning sequence-to-sequence (seq2seq) tasks. While there exists a rich literature that studies classification and regression tasks using solvable models of neural networks, seq2seq tasks have not yet been studied from this perspective. Here, we propose a simple model for a seq2seq task that has the advantage of providing explicit control over the degree of memory, or non-Markovianity, in the sequences -- the stochastic switching-Ornstein-Uhlenbeck (SSOU) model. We introduce a measure of non-Markovianity to quantify the amount of memory in the sequences. For a minimal auto-regressive (AR) learning model trained on this task, we identify two learning regimes corresponding to distinct phases in the stationary state of the SSOU process. These phases emerge from the interplay between two different time scales that govern the sequence statistics. Moreover, we observe that while increasing the integration window of the AR model always improves performance, albeit with diminishing returns, increasing the non-Markovianity of the input sequences can improve or degrade its performance. Finally, we perform experiments with recurrent and convolutional neural networks that show that our observations carry over to more complicated neural network architectures.

arXiv commentCode to reproduce our experiments available at https://github.com/alirezaseif/nonmarkovian_learning

References in corpus (29)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Machine learning and the physical sciences
• Entanglement and non-Markovianity of quantum evolutions
• Measure for the Non-Markovianity of Quantum Processes
• Linking connectivity, dynamics and computations in low-rank recurrent neural networks
• Canonical form of master equations and characterization of non-Markovianity
• Finite-time generalization of the thermodynamic uncertainty relation
• Classification and Geometry of General Perceptual Manifolds
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Estimating entropy production from waiting time distributions
• Prediction and real-time compensation of qubit decoherence via machine-learning
• The Gaussian equivalence of generative models for learning with shallow neural networks
• The interplay between randomness and structure during learning in RNNs
• Manifestations of Projection-Induced Memory: General Theory and the Tilted Single File
• Real-time calibration with spectator qubits
• Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
• Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
• Response functions as quantifiers of non-Markovianity
• Modelling Active Non-Markovian Oscillations
• Probing transfer learning with a model of synthetic correlated datasets
• A Toolbox for Quantifying Memory in Dynamics Along Reaction Coordinates
• Volterra-series approach to stochastic nonlinear dynamics: linear response of the Van der Pol oscillator driven by white noise
• Asymptotics of Ridge (less) Regression under General Source Condition
• Quantifying non-Markovianity via conditional mutual information
• On the interplay between data structure and loss function in classification problems
• Effective bandwidth of non-Markovian packet traffic

Cited by in corpus (1)

• Mapping of attention mechanisms to a generalized Potts model