12 papers
Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it
Hadi Reisizadeh, Jiajun Ruan, Sijia Liu +1
Unlearning in large language models (LLMs) is typically evaluated at the output level, where a model appears to suppress sensitive or undesirable content. In this work, we show tha…
Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
Chongyu Fan, Gaowen Liu, Mingyi Hong +2
Muon is a matrix-aware optimizer that leverages Newton-Schulz (NS) iterations to enforce spectral gradient orthogonalization by driving all singular values of the momentum matrix t…
Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
Athanasios Glentis, Dawei Li, Chung-Yiu Yau +1
It is widely believed that stochastic gradient descent (SGD) performs significantly worse than adaptive optimizers such as Adam in pre-training Large Language Models (LLMs). Yet th…
Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization
Yancheng Huang, Changsheng Wang, Chongyu Fan +7
Foundation models, such as large language models (LLMs), are powerful but often require customization before deployment to satisfy practical constraints such as safety, privacy, an…
Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization
Yicheng Lang, Changsheng Wang, Yihua Zhang +4
Zeroth-order (ZO) optimization provides a gradient-free alternative to first-order (FO) methods by estimating gradients via finite differences of function evaluations, and has rece…
Leak@: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding
Hadi Reisizadeh, Jiajun Ruan, Yiwei Chen +3
Unlearning in large language models (LLMs) is critical for regulatory compliance and for building ethical generative AI systems that avoid producing private, toxic, illegal, or cop…
EPiC: Towards Lossless Speedup for Reasoning Training through Edge-Preserving CoT Condensation
Jinghan Jia, Hadi Reisizadeh, Chongyu Fan +3
Large language models (LLMs) have shown remarkable reasoning capabilities when trained with chain-of-thought (CoT) supervision. However, the long and verbose CoT traces, especially…
BLUR: A Bi-Level Optimization Approach for LLM Unlearning
Hadi Reisizadeh, Jinghan Jia, Zhiqi Bu +6
Enabling large language models (LLMs) to unlearn knowledge and capabilities acquired during training has proven vital for ensuring compliance with data regulations and promoting et…
A Doubly Stochastically Perturbed Algorithm for Linearly Constrained Bilevel Optimization
Prashant Khanduri, Ioannis Tsaknakis, Yihua Zhang +2
In this work, we develop analysis and algorithms for a class of (stochastic) bilevel optimization problems whose lower-level (LL) problem is strongly convex and linearly constraine…
SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models
Anil Ramakrishna, Yixin Wan, Xiaomeng Jin +6
We introduce SemEval-2025 Task 4: unlearning sensitive content from Large Language Models (LLMs). The task features 3 subtasks for LLM unlearning spanning different use cases: (1)…
LUME: LLM Unlearning with Multitask Evaluations
Anil Ramakrishna, Yixin Wan, Xiaomeng Jin +6
Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning b…
Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond
Chongyu Fan, Jinghan Jia, Yihua Zhang +3
The LLM unlearning technique has recently been introduced to comply with data regulations and address the safety and ethical concerns of LLMs by removing the undesired data-model i…