4 papers · 1 filter
Doc-to-LoRA: Learning to Instantly Internalize Contexts
Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka +1
Long input sequences are central to in-context learning, document understanding, and multi-step reasoning of Large Language Models (LLMs). However, the quadratic attention cost of…
Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
Yoav Gelberg, Koshi Eguchi, Takuya Akiba +1
So far, expensive finetuning beyond the pretraining sequence length has been a requirement for effectively extending the context of language models (LM). In this work, we break thi…
ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution
Robert Tjarko Lange, Yuki Imajuku, Edoardo Cetin
We introduce ShinkaEvolve: a new open-source framework leveraging large language models (LLMs) to advance scientific discovery with state-of-the-art performance and unprecedented e…
Large Language Models to Diffusion Finetuning
Edoardo Cetin, Tianyu Zhao, Yujin Tang
We propose a new finetuning method to provide pre-trained large language models (LMs) the ability to scale test-time compute through the diffusion framework. By increasing the numb…