SELF: Self-Extend the Context Length With Logistic Growth Function

Phat Thanh Dang, Saahil Thoppay, Wang Yang +3

Large language models suffer issues when operated on long contexts that are larger than their training context length due to the standard position encoding for tokens in the attent…