3 papers
cs.LG2025
AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Laws
Oren Neumann, Claudius Gros
Neural scaling laws are observed in a range of domains, to date with no universal understanding of why they occur. Recent theories suggest that loss power laws arise from Zipf's la…
cs.LG2025
Reorganizing attention-space geometry with expressive attention
Claudius Gros
Attention regulates information transfer between tokens. For this, query and key vectors are compared, typically in terms of a scalar product, , together wi…
cs.LG2025
Small transformer architectures for task switching
Claudius Gros
The rapid progress seen in terms of large-scale generative AI is largely based on the attention mechanism. It is conversely non-trivial to conceive small-scale applications for whi…