1 citations · 1 across the 9 of their papers we have counts for
9 papers
Structured Recurrent Mixers for Massively Parallelized Sequence Generation
Benjamin L. Badger
Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and infe…
Toeplitz MLP Mixers are Low Complexity, Information-Rich Sequence Models
Benjamin L. Badger, Ethan Roland
Transformer-based large language models are in some respects limited by the quadratic time and space computational complexity of attention. We introduce the Toeplitz MLP Mixer (TMM…
Language Model Memory and Memory Models for Language
Benjamin L. Badger
The ability of machine learning models to store input information in hidden layer vector embeddings, analogous to the concept of `memory', is widely employed but not well character…
Know Your Limits: Entropy Estimation Modeling for Compression and Generalization
Benjamin L. Badger, Matthew Neligeorge
Language prediction is constrained by informational entropy intrinsic to language, such that there exists a limit to how accurate any language model can become and equivalently a l…
Masked Mixers for Language Generation and Retrieval
Benjamin L. Badger
Attention mechanisms that confer selective focus on a strict subset of input elements are nearly ubiquitous in language models today. We posit there to be downside to the use of at…
Adversarial Examples from Dimensional Invariance
Benjamin L. Badger
Adversarial examples have been found for various deep as well as shallow learning models, and have at various times been suggested to be either fixable model-specific bugs, or else…
Why Deep Learning Generalizes
Benjamin L. Badger
Very large deep learning models trained using gradient descent are remarkably resistant to memorization given their huge capacity, but are at the same time capable of fitting large…
Small Language Models for Tabular Data
Benjamin L. Badger
Supervised deep learning is most commonly applied to difficult problems defined on large and often extensively curated datasets. Here we demonstrate the ability of deep representat…
Depth and Representation in Vision Models
Benjamin L. Badger
Deep learning models develop successive representations of their input in sequential layers, the last of which maps the final representation to the output. Here we investigate the…