5 papers
NRGPT: An Energy-based Alternative for GPT
Nima Dehmamy, Benjamin Hoover, Bishwajit Saha +3
Generative Pre-trained Transformer (GPT) architectures are the most popular design for language modeling. Energy-based modeling is a different paradigm that views inference as a dy…
Energy-Based Dynamical Models for Neurocomputation, Learning, and Optimization
Arthur N. Montanari, Francesco Bullo, Dmitry Krotov +1
Recent advances at the intersection of control theory, neuroscience, and machine learning have revealed novel mechanisms by which dynamical systems perform computation. These advan…
Dense Associative Memory with Epanechnikov Energy
Benjamin Hoover, Zhaoyang Shi, Krishnakumar Balasubramanian +2
We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum…
Small Models, Smarter Learning: The Power of Joint Task Training
Csaba Both, Benjamin Hoover, Hendrik Strobelt +4
Multi-task learning improves generalization, but when does it reduce the model capacity required to learn? We provide a systematic study of how joint training affects the learning…
Modern Methods in Associative Memory
Dmitry Krotov, Benjamin Hoover, Parikshit Ram +1
Associative Memories like the famous Hopfield Networks are elegant models for describing fully recurrent neural networks whose fundamental job is to store and retrieve information.…