7.8k citations · 21.1k across the 144 of their papers we have counts for
102 papers · 1 filter
Loss Smoothing for Stable Adaptation Under Distribution Shift
Darshan Patil, Ekaterina Lobacheva, Razvan Pascanu +1
In settings such as fine-tuning and reinforcement learning, neural networks are often adapted under distribution shift. Standard adaptation methods typically optimize the target ob…
Mining Generalizable Activation Functions
Alex Vitvitskyi, Michael Boratko, Matej Grcic +3
The choice of activation function is an active area of research, with different proposals aimed at improving optimization, while maintaining expressivity. Additionally, the activat…
Perplexity Cannot Always Tell Right from Wrong
Petar Veličković, Federico Barbero, Christos Perivolaropoulos +2
Perplexity -- a function measuring a model's overall level of "surprise" when encountering a particular output -- has gained significant traction in recent years, both as a loss fu…
MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling
Mahdi Karami, Ali Behrouz, Peilin Zhong +2
State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurre…
Fine-Tuned In-Context Learners for Efficient Adaptation
Jorg Bornschein, Clare Lyle, Yazhe Li +3
When adapting large language models (LLMs) to a specific downstream task, two primary approaches are commonly employed: (1) prompt engineering, often with in-context few-shot learn…
What Can Grokking Teach Us About Learning Under Nonstationarity?
Clare Lyle, Gharda Sokar, Razvan Pascanu +1
In continual learning problems, it is often necessary to overwrite components of a neural network's learned representation in response to changes in the data stream; however, neura…