Showing cs.LGShow all
3 papers · 1 filter
cs.LG2025
Spark Transformer: Reactivating Sparsity in FFN and Attention
Chong You, Kan Wu, Zhipeng Jia +16
The discovery of the lazy neuron phenomenon in trained Transformers, where the vast majority of neurons in their feed-forward networks (FFN) are inactive for each token, has spurre…
cs.LG2024
Mimetic Initialization Helps State Space Models Learn to Recall
Asher Trockman, Hrayr Harutyunyan, J. Zico Kolter +2
Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with r…
cs.LG2024
HiRE: High Recall Approximate Top- Estimation for Efficient LLM Inference
Yashas Samaga B L, Varun Yerram, Chong You +4
Autoregressive decoding with generative Large Language Models (LLMs) on accelerators (GPUs/TPUs) is often memory-bound where most of the time is spent on transferring model paramet…