Kernel Deformed Exponential Families for Sparse Continuous Attention

Alexander Moreno, Supriya Nagesh, Zhenke Wu +2

Attention mechanisms take an expectation of a data representation with respect to probability weights. This creates summary statistics that focus on important features. Recently, (…