6 papers · 1 filter
On the Limits of Prompt-Conditioned Language Models as General-Purpose Learners
David Mguni, Julian Ma, Jun Wang
Large Language Models (LLMs) are frequently portrayed as general-purpose solvers capable of solving arbitrary tasks. We argue that this view overlooks a fundamental constraint: lan…
Efficient and Principled Scientific Discovery through Bayesian Optimization: A Tutorial
Zhongwei Yu, Rasul Tutunov, Alexandre Max Maraval +13
Traditional scientific discovery relies on an iterative hypothesise-experiment-refine cycle that has driven progress for centuries, but its intuitive, ad-hoc implementation often w…
Bottlenecked Transformers: Periodic KV Cache Consolidation for Generalised Reasoning
Adnan Oomerjee, Zafeirios Fountas, Haitham Bou-Ammar +1
Transformer LLMs have been shown to exhibit strong reasoning ability that scales with inference-time compute, most prominently through token-space "thinking" chains of thought. A g…
SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning
Hugo Hazard, Zafeirios Fountas, Martin A. Benfeghoul +3
Continual learning, one's ability to adapt to a sequence of tasks without forgetting previously acquired knowledge, remains a major challenge in machine learning and a key gap betw…
Subjective Depth and Timescale Transformers: Learning Where and When to Compute
Frederico Wieser, Martin Benfeghoul, Haitham Bou Ammar +2
The rigid, uniform allocation of computation in standard Transformer (TF) architectures can limit their efficiency and scalability, particularly for large-scale models and long seq…
Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods
Martin Benfeghoul, Teresa Delgado, Adnan Oomerjee +3
Transformers' quadratic computational complexity limits their scalability despite remarkable performance. While linear attention reduces this to linear complexity, pre-training suc…