Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs

Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides +3

Modern language models represent text using discrete token-level embeddings, which forces recurring multi-token patterns to be learned implicitly across Transformer layers. Both Ov…