Formal Algorithms for Transformers

Abstract

This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

arXiv comment16 pages, 15 algorithms

Cited by in corpus (5)

• TrAISformer -- A Transformer Network with Sparse Augmented Data Representation and Cross Entropy Loss for AIS-based Vessel Trajectory Prediction
• What Formal Languages Can Transformers Express? A Survey
• RankFormer: Listwise Learning-to-Rank Using Listwide Labels
• Accuracy vs Memory Advantage in the Quantum Simulation of Stochastic Processes
• M&M: Multimodal-Multitask Model Integrating Audiovisual Cues in Cognitive Load Assessment