36 citations · 51 across the 27 of their papers we have counts for
31 papers
OptiSelect: How does the Optimizer Shape Data Curriculum?
Simin Fan, Alireza Abdollahpoorrostam, Martin Jaggi
Online data selection has demonstrated substantial efficiency gains for LLM pretraining by training on the most valuable candidates within each batch. Since a candidate's value is…
Self-Repulsive Sampling for Diffusion Language Models
Michael Helcig, Martin Jaggi
Sampling several responses and voting over their answers can improve a language model's accuracy, but repeated answers limit the benefit of additional samples. Raising temperature…
Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026
Sunnie S. Y. Kim, Wesley Hanwen Deng, Jennifer Wortman Vaughan +8
LLMs are rapidly reshaping peer review, making it important to understand how reviewers use them in practice and how different LLM-use policies affect review outcomes. We investiga…
MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
Simla Burcu Harma, Danila Mishin, Zhengyuan Su +7
4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers. Prior work addresses this problem via data rotation or mixed-…
Benchmarking Optimizers for Large Language Model Pretraining
Andrei Semenov, Matteo Pagliardini, Martin Jaggi
The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Clai…
Semantic uncertainty in advanced decoding methods for LLM generation
Darius Foodeei, Simin Fan, Martin Jaggi
This study investigates semantic uncertainty in large language model (LLM) outputs across different decoding methods, focusing on emerging techniques like speculative sampling and…
Gradient-Normalized Smoothness for Optimization with Approximate Hessians
Andrei Semenov, Martin Jaggi, Nikita Doikov
In this work, we develop new optimization algorithms that use approximate second-order information combined with the gradient regularization technique to achieve fast global conver…
Balancing Knowledge Delivery and Emotional Comfort in Healthcare Conversational Systems
Shang-Chi Tsai, Yun-Nung Chen
With the advancement of large language models, many dialogue systems are now capable of providing reasonable and informative responses to patients' medical conditions. However, whe…
GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
Simin Fan, Maria Ios Glarou, Martin Jaggi
The performance of large language models (LLMs) across diverse downstream applications is fundamentally governed by the quality and composition of their pretraining corpora. Existi…
NeuralGrok: Accelerate Grokking by Neural Gradient Transformation
Xinyu Zhou, Simin Fan, Martin Jaggi +1
Grokking is proposed and widely studied as an intricate phenomenon in which generalization is achieved after a long-lasting period of overfitting. In this work, we propose NeuralGr…
Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs
Thierry Bossy, Julien Vignoud, Tahseen Rabbani +2
Federated learning (FL) is a popular paradigm for collaborative training which avoids direct data exposure between clients. However, data privacy issues still remain: FL-trained la…
Leveraging the true depth of LLMs
Ramón Calvo González, Daniele Paliotta, Matteo Pagliardini +2
The remarkable capabilities of Large Language Models (LLMs) are overshadowed by their immense computational cost. While recent work has shown that many LLM layers can be reordered…
Improving Stochastic Cubic Newton with Momentum
El Mahdi Chayti, Nikita Doikov, Martin Jaggi
We study stochastic second-order methods for solving general non-convex optimization problems. We propose using a special version of momentum to stabilize the stochastic gradient a…
HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimation
Xinyu Zhou, Simin Fan, Martin Jaggi
Influence functions provide a principled method to assess the contribution of individual training samples to a specific target. Yet, their high computational costs limit their appl…
CoBo: Collaborative Learning via Bilevel Optimization
Diba Hashemi, Lie He, Martin Jaggi
Collaborative learning is an important tool to train multiple clients more effectively by enabling communication among clients. Identifying helpful clients, however, presents chall…
On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists
Dongyang Fan, Bettina Messmer, Nikita Doikov +1
On-device LLMs have gained increasing attention for their ability to enhance privacy and provide a personalized user experience. To facilitate private learning with scarce data, Fe…
A New First-Order Meta-Learning Algorithm with Convergence Guarantees
El Mahdi Chayti, Martin Jaggi
Learning new tasks by leveraging prior experience is a fundamental trait of intelligent systems. While Model-Agnostic Meta-Learning (MAML) is a leading approach, it suffers from si…
Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants
Beatriz Borges, Negar Foroutan, Deniz Bayazit +87
AI assistants are being increasingly used by students enrolled in higher education institutions. While these tools provide opportunities for improved teaching and education, they a…
Deep Grokking: Would Deep Neural Networks Generalize Better?
Simin Fan, Razvan Pascanu, Martin Jaggi
Recent research on the grokking phenomenon has illuminated the intricacies of neural networks' training dynamics and their generalization behaviors. Grokking refers to a sharp rise…
Effective Interplay between Sparsity and Quantization: From Theory to Practice
Simla Burcu Harma, Ayan Chakraborty, Elizaveta Kostenok +8
The increasing size of deep neural networks (DNNs) necessitates effective model compression to reduce their computational and memory footprints. Sparsity and quantization are two p…
Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
Alexander Hägele, Elie Bakouch, Atli Kosson +3
Scale has become a main ingredient in obtaining strong machine learning models. As a result, understanding a model's scaling properties is key to effectively designing both the rig…
Personalized Collaborative Fine-Tuning for On-Device Large Language Models
Nicolas Wagner, Dongyang Fan, Martin Jaggi
We explore on-device self-supervised collaborative fine-tuning of large language models with limited local data availability. Taking inspiration from the collaborative learning com…
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci +6
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rot…
Towards an empirical understanding of MoE design choices
Dongyang Fan, Bettina Messmer, Martin Jaggi
In this study, we systematically evaluate the impact of common design choices in Mixture of Experts (MoEs) on validation performance, uncovering distinct influences at token and se…