Intrinsic Entropy of Context Length Scaling in LLMs

Jingzhe Shi, Qinwei Ma, Hongyi Liu +3

Long Context Language Models have drawn great attention in the past few years. There has been work discussing the impact of long context on Language Model performance: some find th…