FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models

Bowen Yang, Jingbo Zhou, Qinghong Miao +1

Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams,…