1.3k citations · 3.8k across the 222 of their papers we have counts for
31 papers · 1 filter
Training Neural Networks at Any Scale
Thomas Pethick, Kimon Antonakopoulos, Antonio Silveti-Falls +2
This article reviews modern optimization methods for training neural networks with an emphasis on efficiency and scale. We present state-of-the-art optimization algorithms under a…
Selective Rotary Position Embedding
Sajad Movahedi, Timur Carstensen, Arshia Afzal +3
Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations…
Ascent Fails to Forget
Ioannis Mavrothalassitis, Pol Puigdemont, Noam Itzhak Levi +1
Contrary to common belief, we show that gradient ascent-based unconstrained optimization methods frequently fail to perform machine unlearning, a phenomenon we attribute to the inh…
Rate optimal learning of equilibria from data
Till Freihaut, Luca Viano, Emanuele Nevali +3
We close open theoretical gaps in Multi-Agent Imitation Learning (MAIL) by characterizing the limits of non-interactive MAIL and presenting the first interactive algorithm with nea…
Accelerating Spectral Clustering under Fairness Constraints
Francesco Tonin, Alex Lambert, Johan A. K. Suykens +1
Fairness of decision-making algorithms is an increasingly important issue. In this paper, we focus on spectral clustering with group fairness constraints, where every demographic g…
BLUR: A Bi-Level Optimization Approach for LLM Unlearning
Hadi Reisizadeh, Jinghan Jia, Zhiqi Bu +6
Enabling large language models (LLMs) to unlearn knowledge and capabilities acquired during training has proven vital for ensuring compliance with data regulations and promoting et…
Diffusion-based Cumulative Adversarial Purification for Vision Language Models
Jia Fu, Yongtao Wu, Yihang Chen +5
Vision Language Models (VLMs) have shown remarkable capabilities in multimodal understanding, yet their susceptibility to adversarial perturbations poses a significant threat to th…
Robustness in Both Domains: CLIP Needs a Robust Text Encoder
Elias Abad Rocamora, Christian Schlarmann, Naman Deep Singh +3
Adversarial input attacks can cause a significant shift of CLIP embeddings. This can affect the downstream robustness of models incorporating CLIP in the pipeline, such as text-to-…
Generalized Gradient Norm Clipping & Non-Euclidean -Smoothness
Thomas Pethick, Wanyun Xie, Mete Erdogan +3
This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The meth…
Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning
Yixin Wan, Anil Ramakrishna, Kai-Wei Chang +2
Large Language Model (LLM) unlearning has recently gained significant attention, driven by the need to remove unwanted information, such as private, sensitive, or copyrighted conte…
Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
Wanyun Xie, Francesco Tonin, Volkan Cevher
Training data mixtures greatly impact the generalization performance of large language models. Existing domain reweighting methods often rely on costly weight computations and requ…
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
Melis Ilayda Bal, Volkan Cevher, Michael Muehlebach
Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introduce Efficient Selective Language Model…
Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games
Yi Feng, Kaito Fujii, Stratis Skoulakis +2
Since Polyak's pioneering work, heavy ball (HB) momentum has been widely studied in minimization. However, its role in min-max games remains largely unexplored. As a key component…
Layer-wise Quantization for Quantized Optimistic Dual Averaging
Anh Duc Nguyen, Ilia Markov, Frank Zhengqing Wu +4
Modern deep neural networks exhibit heterogeneity across numerous layers of various types such as residuals, multi-head attention, etc., due to varying structures (dimensions, acti…
Efficient Large Language Model Inference with Neural Block Linearization
Mete Erdogan, Francesco Tonin, Volkan Cevher
The high inference demands of transformer-based Large Language Models (LLMs) pose substantial challenges in their deployment. To this end, we introduce Neural Block Linearization (…
Learning Equilibria from Data: Provably Efficient Multi-Agent Imitation Learning
Till Freihaut, Luca Viano, Volkan Cevher +2
This paper provides the first expert sample complexity characterization for learning a Nash equilibrium from expert data in Markov Games. We show that a new quantity named the sing…
Hadamard product in deep learning: Introduction, Advances and Challenges
Grigorios G Chrysos, Yongtao Wu, Razvan Pascanu +2
While convolution and self-attention mechanisms have dominated architectural design in deep learning, this survey examines a fundamental yet understudied primitive: the Hadamard pr…
SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models
Anil Ramakrishna, Yixin Wan, Xiaomeng Jin +6
We introduce SemEval-2025 Task 4: unlearning sensitive content from Large Language Models (LLMs). The task features 3 subtasks for LLM unlearning spanning different use cases: (1)…
MT-NAM: An Efficient and Adaptive Model for Epileptic Seizure Detection
Arshia Afzal, Volkan Cevher, Mahsa Shoaran
Enhancing the accuracy and efficiency of machine learning algorithms employed in neural interface systems is crucial for advancing next-generation intelligent therapeutic devices.…
Quantum-PEFT: Ultra parameter-efficient fine-tuning
Toshiaki Koike-Akino, Francesco Tonin, Yongtao Wu +3
This paper introduces Quantum-PEFT that leverages quantum computations for parameter-efficient fine-tuning (PEFT). Unlike other additive PEFT methods, such as low-rank adaptation (…
LUME: LLM Unlearning with Multitask Evaluations
Anil Ramakrishna, Yixin Wan, Xiaomeng Jin +6
Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning b…
Adversarial Training for Defense Against Label Poisoning Attacks
Melis Ilayda Bal, Volkan Cevher, Michael Muehlebach
As machine learning models grow in complexity and increasingly rely on publicly sourced data, such as the human-annotated labels used in training large language models, they become…
Single-pass Detection of Jailbreaking Input in Large Language Models
Leyla Naz Candogan, Yongtao Wu, Elias Abad Rocamora +2
Defending aligned Large Language Models (LLMs) against jailbreaking attacks is a challenging problem, with existing approaches requiring multiple requests or even queries to auxili…
Certified Robustness Under Bounded Levenshtein Distance
Elias Abad Rocamora, Grigorios G. Chrysos, Volkan Cevher
Text classifiers suffer from small perturbations, that if chosen adversarially, can dramatically change the output of the model. Verification methods can provide robustness certifi…