MC-LSTM: Mass-Conserving LSTM

Abstract

The success of Convolutional Neural Networks (CNNs) in computer vision is mainly driven by their strong inductive bias, which is strong enough to allow CNNs to solve vision-related tasks with random weights, meaning without learning. Similarly, Long Short-Term Memory (LSTM) has a strong inductive bias towards storing information over time. However, many real-world systems are governed by conservation laws, which lead to the redistribution of particular quantities -- e.g. in physical and economical systems. Our novel Mass-Conserving LSTM (MC-LSTM) adheres to these conservation laws by extending the inductive bias of LSTM to model the redistribution of those stored quantities. MC-LSTMs set a new state-of-the-art for neural arithmetic units at learning arithmetic operations, such as addition tasks, which have a strong conservation law, as the sum is constant over time. Further, MC-LSTM is applied to traffic forecasting, modelling a pendulum, and a large benchmark dataset in hydrology, where it sets a new state-of-the-art for predicting peak flows. In the hydrology example, we show that MC-LSTM states correlate with real-world processes and are therefore interpretable.

arXiv comment13 pages (8.5 without references) + 17 pages appendix

References in corpus (24)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning in Neural Networks: An Overview
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Spectral Normalization for Generative Adversarial Networks
• Theory-guided Data Science: A New Paradigm for Scientific Discovery from Data
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Variational Inference with Normalizing Flows
• Deep Image Prior
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Discovering physical concepts with neural networks
• Enforcing Analytic Constraints in Neural-Networks Emulating Physical Systems
• Hamiltonian Neural Networks
• Normalizing Flows for Probabilistic Modeling and Inference
• NeuralHydrology -- Interpreting LSTMs in Hydrology
• Full-Capacity Unitary Recurrent Neural Networks
• Achieving Conservation of Energy in Neural Network Emulators for Climate Modeling
• Neural Arithmetic Logic Units
• Symplectic Recurrent Neural Networks
• Lagrangian Neural Networks
• A Powerful Generative Model Using Random Weights for the Deep Image Representation
• Neural Arithmetic Units
• Neural Status Registers