Showing 2024Show all
3 papers · 1 filter
cs.LG2024
Learning and Transferring Sparse Contextual Bigrams with Linear Transformers
Yunwei Ren, Zixuan Wang, Jason D. Lee
Transformers have excelled in natural language modeling and one reason behind this success is their exceptional ability to combine contextual informal and global knowledge. However…
cs.LG2024
Learning Orthogonal Multi-Index Models: A Fine-Grained Information Exponent Analysis
Yunwei Ren, Jason D. Lee
The information exponent ([BAGJ21]) and its extensions -- which are equivalent to the lowest degree in the Hermite expansion of the link function (after a potential label transform…
stat.ML2024
Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot
Zixuan Wang, Stanley Wei, Daniel Hsu +1
The transformer architecture has prevailed in various deep learning settings due to its exceptional capabilities to select and compose structural information. Motivated by these ca…