Sliding Window Recurrences for Sequence Models

Dragos Secrieru, Garyk Brixi, Yoshua Bengio +3

Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrenc…