Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms

Kewei Li, Yanwen Kong, Yiping Xu +4

Since the emergence of research on improving the length extrapolation capabilities of large language models in 2021, some studies have made modifications to the scaling factor in t…