14 citations · 39 across the 15 of their papers we have counts for
Showing 2021Show all
2 papers · 1 filter
cs.LG2021★ 14 cited
Sinkformers: Transformers with Doubly Stochastic Attention
Michael E. Sander, Pierre Ablin, Mathieu Blondel +1
Attention based models such as Transformers involve pairwise interactions between data points, modeled with a learnable attention matrix. Importantly, this attention matrix is norm…
cs.LG2021★ 11 cited
Momentum Residual Neural Networks
Michael E. Sander, Pierre Ablin, Mathieu Blondel +1
The training of deep residual neural networks (ResNets) with backpropagation has a memory cost that increases linearly with respect to the depth of the network. A way to circumvent…