Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Masking

Guojie Liu, Yiqi Wang, Yanfeng Yang +4

Providing extensive context via prompting is vital for leveraging the capabilities of Large Language Models (LLMs). However, lengthy contexts significantly increase inference laten…