4 papers · 1 filter
Towards Distributed Neural Architectures
Aditya Cowsik, Tianyu He, Andrey Gromov
We introduce and train distributed neural architectures (DNA) in vision and language domains. DNAs are initialized with a proto-architecture that consists of (transformer, MLP, att…
The Persian Rug: solving toy models of superposition using large-scale symmetries
Aditya Cowsik, Kfir Dolev, Alex Infanger
We present a complete mechanistic description of the algorithm learned by a minimal non-linear sparse data autoencoder in the limit of large input dimension. The model, originally…
Flatter, faster: scaling momentum for optimal speedup of SGD
Aditya Cowsik, Tankut Can, Paolo Glorioso
Commonly used optimization algorithms often show a trade-off between good generalization and fast training times. For instance, stochastic gradient descent (SGD) tends to have good…
Breast Cancer Diagnosis by Higher-Order Probabilistic Perceptrons
Aditya Cowsik, John W. Clark
A two-layer neural network model that systematically includes correlations among input variables to arbitrary order and is designed to implement Bayes inference has been adapted to…