Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruning

Yiju Guo, Wenkai Yang, Zexu Sun +3

Large language models (LLMs) have demonstrated significant improvements in contextual understanding. However, their ability to attend to truly critical information during long-cont…