1 citations · 1 across the 3 of their papers we have counts for
Showing cs.LGShow all
3 papers · 1 filter
cs.LG2025
Adaptive Computation Pruning for the Forgetting Transformer
Zhixuan Lin, Johan Obando-Ceron, Xu Owen He +1
The recently proposed Forgetting Transformer (FoX) incorporates a forget gate into softmax attention and has shown consistently better or on-par performance compared to the standar…
cs.LG2025
Forgetting Transformer: Softmax Attention with a Forget Gate
Zhixuan Lin, Evgenii Nikishin, Xu Owen He +1
An essential component of modern recurrent sequence models is the forget gate. While Transformers do not have an explicit recurrent form, we show that a forget gate can be naturall…
cs.LG2024
Mixture of A Million Experts
Xu Owen He
The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. Sparse mixt…