Reservoir Transformers
arXiv:2012.15045
Abstract
We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear "reservoir" layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.
arXiv commentACL 2021
References in corpus (6)
- Recent Advances in Physical Reservoir Computing: A Review
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
- Random Feature Attention
- Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers