12 papers
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
Tommaso Cerruti, Tim Rieder, George Rowlands +2
Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context. This paper prese…
POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents
Qiaoyuan Zheng, Yiqu Yang, Qi Gao +1
LLM agents increasingly have access to private user data and act on the user's behalf when interacting with third-party systems. The user defines what may and must not be shared, a…
SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
Alexis Limozin, Eduard Durech, Torsten Hoefler +2
Recent mixed-policy optimization methods for LLM reasoning that interleave or blend supervised and reinforcement learning signals report improvements over the standard SFT-then-RL…
An Engineering Journey Training Large Language Models at Scale on Alps: The Apertus Experience
Jonathan Coles, Stefano Schuppli, Lukas Drescher +20
Large Language Models (LLMs) have surged as a transformative technology for science and society, prompting governments worldwide to pursue sovereign AI capabilities that ensure dat…
Mindstorms in Natural Language-Based Societies of Mind
Mingchen Zhuge, Haozhe Liu, Francesco Faccio +23
Both Minsky's "society of mind" and Schmidhuber's "learning to think" inspire diverse societies of large multimodal neural networks (NNs) that solve problems by interviewing each o…
Benchmarking Concept-Spilling Across Languages in LLMs
Ilia Badanin, Daniil Dzenhaliou, Imanol Schlag
Multilingual Large Language Models (LLMs) exhibit remarkable cross-lingual abilities, yet often exhibit a systematic bias toward the representations from other languages, resulting…