Sliced ReLU attention: Quasi-linear contextual expressivity via sorting

François-Xavier Vialard, Siwan Boufadène

We introduce sliced ReLU attention, a new attention mechanism that departs structurally from both softmax and its approximation alternatives. Instead of applying a nonlinearity to…