7 papers
Protoreasoning in Tiny Transformers
Eduardo Valle, Fergal Reid
We show that tiny transformers can profitably employ a simple form of Chain of Thought, which we call protoreasoning, allowing us to study step-by-step reasoning on ~1M-parameter m…
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
James O' Neill, Fergal Reid
Looped, weight-tied Transformers reduce parameters by reusing a single block, but decoding still stores a separate K/V cache for every recurrence step. We show that this loop-index…
Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters
Wenlong Wang, Fergal Reid
Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic content, or because conditio…
When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer
Mayug Maniparambil, Arjun Karuvally, Terrence Sejnowski +1
Reinforcement learning using verifiable rewards (RLVR) improves LLM reasoning, but the conditions under which it transfers across domains -- and why it does so -- remain under-expl…
Low-Rank Key Value Attention
James O'Neill, Robert Clancy, Mariia Matskevichus +1
The key-value (KV) cache is a primary memory bottleneck in Transformers. We propose Low-Rank Key-Value (LRKV) attention, which reduces KV cache memory by exploiting redundancy acro…
Tiny Recursive Reasoning with Mamba-2 Attention Hybrid
Wenlong Wang, Fergal Reid
Recent work on recursive reasoning models like TRM demonstrates that tiny networks (7M parameters) can achieve strong performance on abstract reasoning tasks through latent recursi…