8 papers
SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging
Shunlong Wu, Hai Lin, Shaoshen Chen +5
Existing KV cache compression methods generally operate on discrete tokens or non-semantic chunks. However, such approaches often lead to semantic fragmentation, where linguistical…
Read As Human: Compressing Context via Parallelizable Close Reading and Skimming
Jiwei Tang, Shilei Liu, Zhicheng Zhang +9
Large Language Models (LLMs) demonstrate exceptional capability across diverse tasks. However, their deployment in long-context scenarios is hindered by two challenges: computation…
RAISE: Reinforced Adaptive Instruction Selection For Large Language Models
Qingsong Lv, Yangning Li, Zihua Lan +8
In the instruction fine-tuning of large language models (LLMs), it is widely recognized that a few high-quality instructions are superior to a large number of low-quality instructi…
From Token to Line: Enhancing Code Generation with a Long-Term Perspective
Tingwei Lu, Yangning Li, Liyuan Wang +6
The emergence of large language models (LLMs) has significantly promoted the development of code generation task, sparking a surge in pertinent literature. Current research is hind…
GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment
Jiwei Tang, Zhicheng Zhang, Shunlong Wu +8
Large Language Models (LLMs) have achieved remarkable performance across a wide range of Natural Language Processing (NLP) tasks. However, in long-context scenarios, they face two…
Teaching According to Talents! Instruction Tuning LLMs with Competence-Aware Curriculum Learning
Yangning Li, Tingwei Lu, Yinghui Li +6
Efficient instruction tuning aims to enhance the ultimate performance of large language models (LLMs) trained on a given instruction dataset. Curriculum learning as a typical data…