Deep Learning: Our Miraculous Year 1990-1991

Abstract

The Deep Learning Artificial Neural Networks (NNs) of our team have revolutionised Machine Learning & AI. Many of the basic ideas behind this revolution were published within the 12 months of our "Annus Mirabilis" 1990-1991 at our lab in TU Munich. Back then, few people were interested. But a quarter century later, NNs based on our "Miraculous Year" were on over 3 billion devices, and used many billions of times per day, consuming a significant fraction of the world's compute. In particular, in 1990-91, we laid foundations of Generative AI, publishing principles of (1) Generative Adversarial Networks for Artificial Curiosity and Creativity (now used for deepfakes), (2) Transformers (the T in ChatGPT - see the 1991 Unnormalized Linear Transformer), (3) Pre-training for deep NNs (see the P in ChatGPT), (4) NN distillation (key for DeepSeek), and (5) recurrent World Models for Reinforcement Learning and Planning in partially observable environments. The year 1991 also marks the emergence of the defining features of (6) LSTM, the most cited AI paper of the 20th century (based on deep residual learning and constant error flow through residual NN connections), and (7) the most cited paper of the 21st century, based on our LSTM-inspired Highway Net that was 10 times deeper than previous feedforward NNs. As of 2025, the two most frequently cited scientific articles of all time (with the most Google Scholar citations within 3 years - manuals excluded) are both directly based on our 1991 work.

52 pages, over 300 references, 38 illustrations, extending v1 of 4 Oct 2019

References in corpus (18)

• Deep Learning in Neural Networks: An Overview
• Neural Machine Translation by Jointly Learning to Align and Translate
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Continuous control with deep reinforcement learning
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Playing Atari with Deep Reinforcement Learning
• Deep Residual Learning for Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• Recurrent World Models Facilitate Policy Evolution
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• A Clockwork RNN
• Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation
• Using Fast Weights to Attend to the Recent Past
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Learning to Reason with Third-Order Tensor Products