Showing 2024Show all
2 papers · 1 filter
cs.LG2024
On the Benefits of Rank in Attention Layers
Noah Amsel, Gilad Yehudai, Joan Bruna
Attention-based mechanisms are widely used in machine learning, most prominently in transformers. However, hyperparameters such as the rank of the attention matrices and the number…
cs.DS2024
Fixed-sparsity matrix approximation from matrix-vector products
Noah Amsel, Tyler Chen, Feyza Duman Keles +3
We study the problem of approximating a matrix with a matrix that has a fixed sparsity pattern (e.g., diagonal, banded, etc.), when is accessed only by ma…