6 papers
Self-Play Pretraining with Zero Data
Aditya Cowsik, Kfir Dolev, Michael Y. Li +4
Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model's behalf. A more general appro…
Hierarchical Prototype Emergence in Modern Hopfield Models
Aditya Cowsik, Adithya Sriram
Hierarchical correlations are a universal feature of any realistic model of data, and the question of how associative memory models may learn these correlations and generalize beyo…
Towards Distributed Neural Architectures
Aditya Cowsik, Tianyu He, Andrey Gromov
We introduce and train distributed neural architectures (DNA) in vision and language domains. DNAs are initialized with a proto-architecture that consists of (transformer, MLP, att…
The Persian Rug: solving toy models of superposition using large-scale symmetries
Aditya Cowsik, Kfir Dolev, Alex Infanger
We present a complete mechanistic description of the algorithm learned by a minimal non-linear sparse data autoencoder in the limit of large input dimension. The model, originally…
Flatter, faster: scaling momentum for optimal speedup of SGD
Aditya Cowsik, Tankut Can, Paolo Glorioso
Commonly used optimization algorithms often show a trade-off between good generalization and fast training times. For instance, stochastic gradient descent (SGD) tends to have good…
Breast Cancer Diagnosis by Higher-Order Probabilistic Perceptrons
Aditya Cowsik, John W. Clark
A two-layer neural network model that systematically includes correlations among input variables to arbitrary order and is designed to implement Bayes inference has been adapted to…