Compressing Sequences in the Latent Embedding Space: $K$-Token Merging for Large Language Models

Zihao Xu, John Harvill, Ziwei Fan +3

Large Language Models (LLMs) incur significant computational and memory costs when processing long prompts, as full self-attention scales quadratically with input length. Token com…