4 citations · 8 across the 16 of their papers we have counts for
12 papers · 1 filter
Extracting Forgotten Prompts from Targeted Unlearned Models
Au Ashley Hoi-Ting, Meghdad Kurmanji, William F. Shen +2
Recent unlearning methods (e.g. NPO, DPO, LUNAR) make use of refusal alignment to suppress forgotten data. However, it has been shown that refusal responses might leave traces of u…
The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
Alex Iacob, Andrej Jovanović, William F. Shen +12
Self-improving agents are state-of-the-art on agentic coding benchmarks, yet their search methods assume a stationary evaluation criterion. This ignores a central feature of evolut…
LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
Andrej Jovanović, Alex Iacob, Mher Safaryan +6
Distributed training of foundation models via is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they…
MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
Alex Iacob, Andrej Jovanovic, Mher Safaryan +6
Training large models with distributed data parallelism (DDP) requires frequent communication of gradients across workers, which can saturate bandwidth. Infrequent communication st…
AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling
Preslav Aleksandrov, Meghdad Kurmanji, Fernando Garcia Redondo +7
We introduce the Autoregressive Block-Based Iterative Encoder (AbbIE), a novel recursive generalization of the encoder-only Transformer architecture, which achieves better perplexi…
DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
Alex Iacob, Lorenzo Sani, Mher Safaryan +8
Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synch…
Editing as Unlearning: Are Knowledge Editing Methods Strong Baselines for Large Language Model Unlearning?
Zexi Li, Xiangzhu Wang, William F. Shen +5
Large language Model (LLM) unlearning, i.e., selectively removing information from LLMs, is vital for responsible model deployment. Differently, LLM knowledge editing aims to modif…
LLM Unlearning via Neural Activation Redirection
William F. Shen, Xinchi Qiu, Meghdad Kurmanji +5
The ability to selectively remove knowledge from LLMs is highly desirable. However, existing methods often struggle with balancing unlearning efficacy and retain model utility, and…
DEPT: Decoupled Embeddings for Pre-training Language Models
Alex Iacob, Lorenzo Sani, Meghdad Kurmanji +5
Language Model pre-training uses broad data mixtures to enhance performance across domains and languages. However, training on such heterogeneous text corpora requires extensive an…
How Data Inter-connectivity Shapes LLMs Unlearning: A Structural Unlearning Perspective
Xinchi Qiu, William F. Shen, Yihong Chen +4
While unlearning knowledge from large language models (LLMs) is receiving increasing attention, one important aspect remains unexplored. Existing approaches and benchmarks assume d…
Worldwide Federated Training of Language Models
Alex Iacob, Lorenzo Sani, Bill Marino +3
The reliance of language model training on massive amounts of computation and vast datasets scraped from potentially low-quality, copyrighted, or sensitive data has come into quest…
The Future of Large Language Model Pre-training is Federated
Lorenzo Sani, Alex Iacob, Zeyu Cao +8
Generative pre-trained large language models (LLMs) have demonstrated impressive performance over a wide range of tasks, thanks to the unprecedented amount of data they have been t…