5 papers
Structured Recurrent Mixers for Massively Parallelized Sequence Generation
Benjamin L. Badger
Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and infe…
Language Model Memory and Memory Models for Language
Benjamin L. Badger
The ability of machine learning models to store input information in hidden layer vector embeddings, analogous to the concept of `memory', is widely employed but not well character…
Toeplitz MLP Mixers are Low Complexity, Information-Rich Sequence Models
Benjamin L. Badger, Ethan Roland
Transformer-based large language models are in some respects limited by the quadratic time and space computational complexity of attention. We introduce the Toeplitz MLP Mixer (TMM…
Know Your Limits: Entropy Estimation Modeling for Compression and Generalization
Benjamin L. Badger, Matthew Neligeorge
Language prediction is constrained by informational entropy intrinsic to language, such that there exists a limit to how accurate any language model can become and equivalently a l…
Masked Mixers for Language Generation and Retrieval
Benjamin L. Badger
Attention mechanisms that confer selective focus on a strict subset of input elements are nearly ubiquitous in language models today. We posit there to be downside to the use of at…