23 citations · 38 across the 10 of their papers we have counts for
Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Nested Sparsity for Scalable Pretraining and Continual Learning
Tianzhe Chu, Druv Pai, Yi Ma
Mixture-of-experts (MoE) transformers, the standard backbone for modern large-scale deep learning, employ conditional computation: only a sparse subset of model weights is used in…
cs.LG2023★ 4 cited
White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?
Yaodong Yu, Sam Buchanan, Druv Pai +7
In this paper, we contend that a natural objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a low-dimension…
cs.LG2023★ 23 cited
White-Box Transformers via Sparse Rate Reduction
Yaodong Yu, Sam Buchanan, Druv Pai +5
In this paper, we contend that the objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a mixture of low-dime…