Publications (174)
Simple Unsupervised Keyphrase Extraction using Sentence Embeddings
Kamil Bennani-Smires, Claudiu Musat, Andreea Hossmann +2
Keyphrase extraction is the task of automatically selecting a small set of phrases that best describe a given free text document. Supervised keyphrase extraction requires large amo…
Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants
Beatriz Borges, Negar Foroutan, Deniz Bayazit +87
AI assistants are being increasingly used by students enrolled in higher education institutions. While these tools provide opportunities for improved teaching and education, they a…
Byzantine-Robust Decentralized Learning via ClippedGossip
Lie He, Sai Praneeth Karimireddy, Martin Jaggi
In this paper, we study the challenging task of Byzantine-robust decentralized training on arbitrary communication graphs. Unlike federated learning where workers communicate throu…
Faster Coordinate Descent via Adaptive Importance Sampling
Dmytro Perekrestenko, Volkan Cevher, Martin Jaggi
Coordinate descent methods employ random partial updates of decision variables in order to solve huge-scale convex optimization problems. In this work, we introduce new adaptive ru…
Exact Optimization of Conformal Predictors via Incremental and Decremental Learning
Giovanni Cherubin, Konstantinos Chatzikokolakis, Martin Jaggi
Conformal Predictors (CP) are wrappers around ML models, providing error guarantees under weak assumptions on the data distribution. They are suitable for a wide range of problems,…
A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe
Francesco Locatello, Rajiv Khanna, Michael Tschannen +1
Two of the most fundamental prototypes of greedy optimization are the matching pursuit and Frank-Wolfe algorithms. In this paper, we take a unified view on both classes of methods,…
Taming GANs with Lookahead-Minmax
Tatjana Chavdarova, Matteo Pagliardini, Sebastian U. Stich +2
Generative Adversarial Networks are notoriously challenging to train. The underlying minmax optimization is highly susceptible to the variance of the stochastic gradient and the ro…
PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
Thijs Vogels, Sai Praneeth Karimireddy, Martin Jaggi
Lossy gradient compression has become a practical tool to overcome the communication bottleneck in centrally coordinated distributed training of machine learning models. However, a…
Sparse Communication for Training Deep Networks
Negar Foroutan Eghlidi, Martin Jaggi
Synchronous stochastic gradient descent (SGD) is the most common method used for distributed training of deep learning models. In this algorithm, each worker shares its local gradi…
Robust Cross-lingual Embeddings from Parallel Sentences
Ali Sabet, Prakhar Gupta, Jean-Baptiste Cordonnier +2
Recent advances in cross-lingual word embeddings have primarily relied on mapping-based methods, which project pretrained word embeddings from different languages into a shared spa…
-LoRA: Effective Fine-Tuning via Base Model Rescaling
Aymane El Firdoussi, El Mahdi Chayti, Mohamed El Amine Seddik +1
Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches…
DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret +1
The transformer architecture by Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. W…
Irreducible Curriculum for Language Model Pretraining
Simin Fan, Martin Jaggi
Automatic data selection and curriculum design for training large language models is challenging, with only a few existing methods showing improvements over standard training. Furt…
SKILL: Structured Knowledge Infusion for Large Language Models
Fedor Moiseev, Zhe Dong, Enrique Alfonseca +1
Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. However, it is largely unexplored whether they can better inter…
Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimation
Futong Liu, Tao Lin, Martin Jaggi
Over-parameterized deep neural networks are able to achieve excellent training accuracy while maintaining a small generalization error. It has also been found that they are able to…
Secure Byzantine-Robust Machine Learning
Lie He, Sai Praneeth Karimireddy, Martin Jaggi
Increasingly machine learning systems are being deployed to edge servers and devices (e.g. mobile phones) and trained in a collaborative manner. Such distributed/federated/decentra…
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci +6
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rot…
Deep Grokking: Would Deep Neural Networks Generalize Better?
Simin Fan, Razvan Pascanu, Martin Jaggi
Recent research on the grokking phenomenon has illuminated the intricacies of neural networks' training dynamics and their generalization behaviors. Grokking refers to a sharp rise…
Optimal Model Averaging: Towards Personalized Collaborative Learning
Felix Grimberg, Mary-Anne Hartley, Sai P. Karimireddy +1
In federated learning, differences in the data or objectives between the participating nodes motivate approaches to train a personalized machine learning model for each node. One s…
Model Fusion via Optimal Transport
Sidak Pal Singh, Martin Jaggi
Combining different models is a widely used paradigm in machine learning applications. While the most common approach is to form an ensemble of models and average their individual…
A Split-Client Approach to Second-Order Optimization
El Mahdi Chayti, Martin Jaggi
Second-order optimization methods offer superior convergence rates but are often bottlenecked by the wall-clock cost of Hessian computation and factorization. In the moderate-dimen…
Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
Bettina Messmer, Vinko SabolÄec, Martin Jaggi
Dataset curation has become a basis for strong large language model (LLM) performance. While various rule-based filtering heuristics exist for English and multilingual datasets, mo…
Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains
Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish +4
Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. To deepen our understanding of their sequential modeling…
Effective Interplay between Sparsity and Quantization: From Theory to Practice
Simla Burcu Harma, Ayan Chakraborty, Elizaveta Kostenok +8
The increasing size of deep neural networks (DNNs) necessitates effective model compression to reduce their computational and memory footprints. Sparsity and quantization are two p…
On Linear Learning with Manycore Processors
Eliza Wszola, Celestine Mendler-Dünner, Martin Jaggi +1
A new generation of manycore processors is on the rise that offers dozens and more cores on a chip and, in a sense, fuses host processor and accelerator. In this paper we target th…
Evaluating the Search Phase of Neural Architecture Search
Kaicheng Yu, Christian Sciuto, Martin Jaggi +2
Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and val…
Characterizing & Finding Good Data Orderings for Fast Convergence of Sequential Gradient Methods
Amirkeivan Mohtashami, Sebastian Stich, Martin Jaggi
While SGD, which samples from the data with replacement is widely studied in theory, a variant called Random Reshuffling (RR) is more common in practice. RR iterates through random…
Sparsified SGD with Memory
Sebastian U. Stich, Jean-Baptiste Cordonnier, Martin Jaggi
Huge scale machine learning problems are nowadays tackled by distributed optimization algorithms, i.e. algorithms that leverage the compute power of many devices for training. The…
Leveraging Large Amounts of Weakly Supervised Data for Multi-Language Sentiment Classification
Jan Deriu, Aurelien Lucchi, Valeria De Luca +5
This paper presents a novel approach for multi-lingual sentiment classification in short texts. This is a challenging task as the amount of training data in languages other than En…
TiMoE: Time-Aware Mixture of Language Experts
Robin Faro, Dongyang Fan, Tamar Alphaidze +1
Large language models (LLMs) are typically trained on fixed snapshots of the web, which means that their knowledge becomes stale and their predictions risk temporal leakage: relyin…
DoGE: Domain Reweighting with Generalization Estimation
Simin Fan, Matteo Pagliardini, Martin Jaggi
The coverage and composition of the pretraining data significantly impacts the generalization ability of Large Language Models (LLMs). Despite its importance, recent LLMs still rel…
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
Anastasia Koloskova, Sebastian U. Stich, Martin Jaggi
We consider decentralized stochastic optimization with the objective function (e.g. data samples for machine learning task) being distributed over machines that can only commun…
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
Prakhar Gupta, Martin Jaggi
The advent of contextual word embeddings -- representations of words which incorporate semantic and syntactic information from their context -- has led to tremendous improvements o…
Provably Personalized and Robust Federated Learning
Mariel Werner, Lie He, Michael Jordan +2
Identifying clients with similar objectives and learning a model-per-cluster is an intuitive and interpretable approach to personalization in federated learning. However, doing so…
Consensus Control for Decentralized Deep Learning
Lingjing Kong, Tao Lin, Anastasia Koloskova +2
Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. Experiments in earlier works reveal…
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels +2
The performance of optimizers, particularly in deep learning, depends considerably on their chosen hyperparameter configuration. The efficacy of optimizers is often studied under n…
On the Relationship between Self-Attention and Convolutional Layers
Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi
Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping C…
Interpretable Structure-aware Document Encoders with Hierarchical Attention
Khalil Mrini, Claudiu Musat, Michael Baeriswyl +1
We propose a method to create document representations that reflect their internal structure. We modify Tree-LSTMs to hierarchically merge basic elements such as words and sentence…
RelaySum for Decentralized Deep Learning on Heterogeneous Data
Thijs Vogels, Lie He, Anastasia Koloskova +4
In decentralized machine learning, workers compute model updates on their local data. Because the workers only communicate with few neighbors without central coordination, these up…
Learning from History for Byzantine Robust Optimization
Sai Praneeth Karimireddy, Lie He, Martin Jaggi
Byzantine robustness has received significant attention recently given its importance for distributed and federated learning. In spite of this, we identify severe flaws in existing…
Crosslingual Document Embedding as Reduced-Rank Ridge Regression
Martin Josifoski, Ivan S. Paskov, Hristo S. Paskov +2
There has recently been much interest in extending vector-based word representations to multiple languages, such that words can be compared across languages. In this paper, we shif…
A Distributed Second-Order Algorithm You Can Trust
Celestine Dünner, Aurelien Lucchi, Matilde Gargiani +3
Due to the rapid growth of data and computational resources, distributed optimization has become an active research area in recent years. While first-order methods seem to dominate…
MLSys: The New Frontier of Machine Learning Systems
Alexander Ratner, Dan Alistarh, Gustavo Alonso +66
Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains…
Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping
Hsun-Yu Kuo, El Mahdi Chayti, Patrik Reizinger +2
Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length algorithmic tasks. Although they can exhibit strong l…
Unsupervised Scalable Representation Learning for Multivariate Time Series
Jean-Yves Franceschi, Aymeric Dieuleveut, Martin Jaggi
Time series constitute a challenging data type for machine learning algorithms, due to their highly variable lengths and sparse labeling in practice. In this paper, we tackle this…
Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning
Tehila Dahan, Bassel Hamoud, Roie Reshef +2
Communication overhead is a crucial bottleneck in scalable distributed learning. While existing methods aim to efficiently utilize data points, such as Local SGD, Minibatch SGD, an…
Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
Alexander Hägele, Elie Bakouch, Atli Kosson +3
Scale has become a main ingredient in obtaining strong machine learning models. As a result, understanding a model's scaling properties is key to effectively designing both the rig…
Linearly Convergent Frank-Wolfe with Backtracking Line-Search
Fabian Pedregosa, Geoffrey Negiar, Armin Askari +1
Structured constraints in Machine Learning have recently brought the Frank-Wolfe (FW) family of algorithms back in the spotlight. While the classical FW algorithm has poor local co…
Scalable Collaborative Learning via Representation Sharing
Frédéric Berdoz, Abhishek Singh, Martin Jaggi +1
Privacy-preserving machine learning has become a key conundrum for multi-party artificial intelligence. Federated learning (FL) and Split Learning (SL) are two frameworks that enab…
Modular Clinical Decision Support Networks (MoDN) -- Updatable, Interpretable, and Portable Predictions for Evolving Clinical Environments
Cécile Trottet, Thijs Vogels, Martin Jaggi +1
Data-driven Clinical Decision Support Systems (CDSS) have the potential to improve and standardise care with personalised probabilistic guidance. However, the size of data required…
On Convergence of Incremental Gradient for Non-Convex Smooth Functions
Anastasia Koloskova, Nikita Doikov, Sebastian U. Stich +1
In machine learning and neural network optimization, algorithms like incremental gradient, and shuffle SGD are popular due to minimizing the number of cache misses and good practic…
Controllable Topic-Focused Abstractive Summarization
Seyed Ali Bahrainian, Martin Jaggi, Carsten Eickhoff
Controlled abstractive summarization focuses on producing condensed versions of a source article to cover specific aspects by shifting the distribution of generated text towards a…
Overcoming Multi-Model Forgetting
Yassine Benyahia, Kaicheng Yu, Kamil Bennani-Smires +4
We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performan…
NeuralGrok: Accelerate Grokking by Neural Gradient Transformation
Xinyu Zhou, Simin Fan, Martin Jaggi +1
Grokking is proposed and widely studied as an intricate phenomenon in which generalization is achieved after a long-lasting period of overfitting. In this work, we propose NeuralGr…
Second-order optimization with lazy Hessians
Nikita Doikov, El Mahdi Chayti, Martin Jaggi
We analyze Newton's method with lazy Hessian updates for solving general possibly non-convex optimization problems. We propose to reuse a previously seen Hessian for several iterat…
Special Properties of Gradient Descent with Large Learning Rates
Amirkeivan Mohtashami, Martin Jaggi, Sebastian Stich
When training neural networks, it has been widely observed that a large step size is essential in stochastic gradient descent (SGD) for obtaining superior models. However, the effe…
A Combinatorial Algorithm to Compute Regularization Paths
Bernd Gärtner, Joachim Giesen, Martin Jaggi +1
For a wide variety of regularization methods, algorithms computing the entire solution path have been developed recently. Solution path algorithms do not only compute the solution…
Extrapolation for Large-batch Training in Deep Learning
Tao Lin, Lingjing Kong, Sebastian U. Stich +1
Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fra…
Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
Matteo Pagliardini, Prakhar Gupta, Martin Jaggi
The recent tremendous success of unsupervised word embeddings in a multitude of applications raises the obvious question if similar methods could be derived to improve embeddings (…
Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training
Atli Kosson, Bettina Messmer, Martin Jaggi
Learning Rate Warmup is a popular heuristic for training neural networks, especially at larger batch sizes, despite limited understanding of its benefits. Warmup decreases the upda…
Adding vs. Averaging in Distributed Primal-Dual Optimization
Chenxin Ma, Virginia Smith, Martin Jaggi +3
Distributed optimization methods for large-scale machine learning suffer from a communication bottleneck. It is difficult to reduce this bottleneck while still efficiently and accu…
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
Alexander Hägele, Alejandro Hernández-Cano, Atli Kosson +1
Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object. Yet every weight matrix carries two distinct quantitie…
Towards Fully FP8 GEMM LLM Training at Scale
Alejandro Hernández-Cano, Dhia Garbaya, Imanol Schlag +1
Despite the significant potential of FP8 data formats for large language model (LLM) pre-training, their adoption has been limited due to challenges in maintaining stability at sca…
Layer-wise Linear Mode Connectivity
Linara Adilova, Maksym Andriushchenko, Michael Kamp +2
Averaging neural network parameters is an intuitive method for fusing the knowledge of two independent models. It is most prominently used in federated learning. If models are aver…
An Optimal Affine Invariant Smooth Minimization Algorithm
Alexandre d'Aspremont, Cristóbal Guzmán, Martin Jaggi
We formulate an affine invariant implementation of the accelerated first-order algorithm in Nesterov (1983). Its complexity bound is proportional to an affine invariant regularity…
Learning Aerial Image Segmentation from Online Maps
Pascal Kaiser, Jan Dirk Wegner, Aurelien Lucchi +3
This study deals with semantic segmentation of high-resolution (aerial) images where a semantic class label is assigned to each pixel via supervised classification as a basis for a…
Efficient Use of Limited-Memory Accelerators for Linear Learning on Heterogeneous Systems
Celestine Dünner, Thomas Parnell, Martin Jaggi
We propose a generic algorithmic building block to accelerate training of machine learning models on heterogeneous compute systems. Our scheme allows to efficiently employ compute…
Beyond spectral gap: The role of the topology in decentralized learning
Thijs Vogels, Hadrien Hendrikx, Martin Jaggi
In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rat…
Personalized Collaborative Fine-Tuning for On-Device Large Language Models
Nicolas Wagner, Dongyang Fan, Martin Jaggi
We explore on-device self-supervised collaborative fine-tuning of large language models with limited local data availability. Taking inspiration from the collaborative learning com…
Federated Learning for Malware Detection in IoT Devices
Valerian Rey, Pedro Miguel Sánchez Sánchez, Alberto Huertas Celdrán +2
This work investigates the possibilities enabled by federated learning concerning IoT malware detection and studies security issues inherent to this new learning paradigm. In this…
Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs
Thierry Bossy, Julien Vignoud, Tahseen Rabbani +2
Federated learning (FL) is a popular paradigm for collaborative training which avoids direct data exposure between clients. However, data privacy issues still remain: FL-trained la…
Stochastic Difference-of-Convex Optimization with Momentum
El Mahdi Chayti, Martin Jaggi
Stochastic difference-of-convex (DC) optimization is prevalent in numerous machine learning applications, yet its convergence properties under small batch sizes remain poorly under…
Benchmarking Optimizers for Large Language Model Pretraining
Andrei Semenov, Matteo Pagliardini, Martin Jaggi
The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Clai…
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
Guilherme Penedo, Hynek KydlÃÄek, Vinko SabolÄec +7
Pre-training state-of-the-art large language models (LLMs) requires vast amounts of clean and diverse text data. While the open development of large high-quality English pre-traini…
MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
Simla Burcu Harma, Danila Mishin, Zhengyuan Su +7
4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers. Prior work addresses this problem via data rotation or mixed-…
Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
Martin Jaggi
Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count -…
Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection
Yassine Turki, Vinko SabolÄec, Bettina Messmer +1
As Large Language Models (LLMs) scale, data curation has shifted from maximizing volume to optimizing the signal-to-noise ratio by performing quality filtering. However, for many l…
Global linear convergence of Newton's method without strong-convexity or Lipschitz gradients
Sai Praneeth Karimireddy, Sebastian U. Stich, Martin Jaggi
We show that Newton's method converges globally at a linear rate for objective functions whose Hessians are stable. This class of problems includes many functions which are not str…
Forecasting intracranial hypertension using multi-scale waveform metrics
Matthias Hüser, Adrian Kündig, Walter Karlen +2
Objective: Acute intracranial hypertension is an important risk factor of secondary brain damage after traumatic brain injury. Hypertensive episodes are often diagnosed reactively,…
CoBo: Collaborative Learning via Bilevel Optimization
Diba Hashemi, Lie He, Martin Jaggi
Collaborative learning is an important tool to train multiple clients more effectively by enabling communication among clients. Identifying helpful clients, however, presents chall…
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
Thijs Vogels, Sai Praneeth Karimireddy, Martin Jaggi
We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compr…
Multiplication-Free Transformer Training via Piecewise Affine Operations
Atli Kosson, Martin Jaggi
Multiplications are responsible for most of the computational cost involved in neural network training and inference. Recent research has thus looked for ways to reduce the cost as…
Multi-Head Attention: Collaborate Instead of Concatenate
Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi
Attention layers are widely used in natural language processing (NLP) and are beginning to influence computer vision architectures. Training very large transformer models allowed s…
Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates
Sebastian U. Stich, Amirkeivan Mohtashami, Martin Jaggi
It has been experimentally observed that the efficiency of distributed training with stochastic gradient (SGD) depends decisively on the batch size and -- in asynchronous implement…
URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training
Dongyang Fan, Vinko SabolÄec, Martin Jaggi
Large Language Models (LLMs) are commonly pretrained on vast corpora of text without utilizing contextual metadata such as source, quality, or topic, leading to a context-free lear…
Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr +1
We study two factors in neural network training: data parallelism and sparsity; here, data parallelism means processing training data in parallel using distributed systems (or equi…
Pursuits in Structured Non-Convex Matrix Factorizations
Rajiv Khanna, Michael Tschannen, Martin Jaggi
Efficiently representing real world data in a succinct and parsimonious manner is of central importance in many fields. We present a generalized greedy pursuit framework, allowing…
Landmark Attention: Random-Access Infinite Context Length for Transformers
Amirkeivan Mohtashami, Martin Jaggi
While Transformers have shown remarkable success in natural language processing, their attention mechanism's large memory requirements have limited their ability to handle longer c…
An Exponential Lower Bound on the Complexity of Regularization Paths
Bernd Gärtner, Martin Jaggi, Clément Maria
For a variety of regularized optimization problems in machine learning, algorithms computing the entire solution path have been developed recently. Most of these methods are quadra…
An Affine Invariant Linear Convergence Analysis for Frank-Wolfe Algorithms
Simon Lacoste-Julien, Martin Jaggi
We study the linear convergence of variants of the Frank-Wolfe algorithms for some classes of strongly convex problems, using only affine-invariant quantities. As in Guelat & Marco…
LASER: Linear Compression in Wireless Distributed Optimization
Ashok Vardhan Makkuva, Marco Bondaschi, Thijs Vogels +3
Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its pe…
Correlating Twitter Language with Community-Level Health Outcomes
Arno Schneuwly, Ralf Grubenmann, Séverine Rion Logean +2
We study how language on social media is linked to diseases such as atherosclerotic heart disease (AHD), diabetes and various types of cancer. Our proposed model leverages state-of…
An Engineering Journey Training Large Language Models at Scale on Alps: The Apertus Experience
Jonathan Coles, Stefano Schuppli, Lukas Drescher +20
Large Language Models (LLMs) have surged as a transformative technology for science and society, prompting governments worldwide to pursue sovereign AI capabilities that ensure dat…
Towards an empirical understanding of MoE design choices
Dongyang Fan, Bettina Messmer, Martin Jaggi
In this study, we systematically evaluate the impact of common design choices in Mixture of Experts (MoEs) on validation performance, uncovering distinct influences at token and se…
DISCO: A Browser-Based Privacy-Preserving Framework for Distributed Collaborative Learning
Julien T. T. Vignoud, Valérian Rousset, Hugo El Guedj +28
Data is often impractical to share for a range of well considered reasons, such as concerns over privacy, intellectual property, and legal constraints. This not only fragments the…
FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings
Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers +21
Federated Learning (FL) is a novel approach enabling several clients holding sensitive data to collaboratively train machine learning models, without centralizing data. The cross-s…
On Matching Pursuit and Coordinate Descent
Francesco Locatello, Anant Raj, Sai Praneeth Karimireddy +4
Two popular examples of first-order optimization methods over linear spaces are coordinate descent and matching pursuit algorithms, with their randomized variants. While the former…
Approximate Steepest Coordinate Descent
Sebastian U. Stich, Anant Raj, Martin Jaggi
We propose a new selection rule for the coordinate selection in coordinate descent methods for huge-scale optimization. The efficiency of this novel scheme is provably better than…
Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
Atli Kosson, Bettina Messmer, Martin Jaggi
This study investigates how weight decay affects the update behavior of individual neurons in deep neural networks through a combination of applied analysis and experimentation. We…
WAFFLE: Weighted Averaging for Personalized Federated Learning
Martin Beaussart, Felix Grimberg, Mary-Anne Hartley +1
In federated learning, model personalization can be a very effective strategy to deal with heterogeneous training data across clients. We introduce WAFFLE (Weighted Averaging For F…