2 papers
cs.LG2024
LoRA vs Full Fine-tuning: An Illusion of Equivalence
Reece Shuttleworth, Jacob Andreas, Antonio Torralba +1
Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effective…
cs.CL2023
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models
Shikhar Murty, Pratyusha Sharma, Jacob Andreas +1
Recursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently…