Orthogonality Constrained Multi-Head Attention For Keyword Spotting

Mingu Lee, Jinkyu Lee, Hye Jin Jang +3

Multi-head attention mechanism is capable of learning various representations from sequential data while paying attention to different subsequences, e.g., word-pieces or syllables…