3 papers
cs.LG2026
Fine-Tuning Dynamics of In-Context Factual Recall in Transformers
Ruomin Huang, Eshaan Nichani, Jason D. Lee +1
In-context learning \ -- performing tasks based on examples given in the prompt \ -- is an important capability that has emerged in large language models and has received significa…
cs.LG2024
Task Diversity Shortens the ICL Plateau
Jaeyeon Kim, Sehyun Kwon, Joo Young Choi +4
In-context learning (ICL) describes a language model's ability to generate outputs based on a set of input demonstrations and a subsequent query. To understand this remarkable capa…
cs.LG2024
LoRA Training in the NTK Regime has No Spurious Local Minima
Uijeong Jang, Jason D. Lee, Ernest K. Ryu
Low-rank adaptation (LoRA) has become the standard approach for parameter-efficient fine-tuning of large language models (LLM), but our theoretical understanding of LoRA has been l…