Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs

Maria Lomeli, Antoine Groudiev, Matthijs Douze +7

This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference…