17 citations · 39 across the 35 of their papers we have counts for
37 papers
Learning to Read Out: Unembedding Dynamics in Language Model Pretraining
Matteo He, William F. Shen, Alex Iacob +3
When does a language model acquire a capability? Developmental studies usually answer this by tracking when models start producing the correct token. But that prediction passes thr…
Extracting Forgotten Prompts from Targeted Unlearned Models
Au Ashley Hoi-Ting, Meghdad Kurmanji, William F. Shen +2
Recent unlearning methods (e.g. NPO, DPO, LUNAR) make use of refusal alignment to suppress forgotten data. However, it has been shown that refusal responses might leave traces of u…
Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens
Matteo He, William F. Shen, Xinchi Qiu +1
A language model's prediction of its next token develops across layers, and lens methods track this process by decoding intermediate hidden states into tokens. But a lens reading r…
Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment
Yan Gao, Mohammad Naseri, Javier Fernandez-Marques +19
Federated learning (FL) has emerged as a key approach for training models across decentralized data, yet benchmarking in FL remains difficult to reproduce, compare, and extend. Exi…
How LLM Task-Adaptation Reshapes Alignment: A Multi-dimensional Study of Behavioral and Representational Drift
James Elcock, William F. Shen, Xinchi Qiu +1
Post-training is a key mechanism for adapting large language models to downstream tasks. While prior work suggests that task adaptation can alter a model's pre-existing alignment,…
The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
Alex Iacob, Andrej Jovanović, William F. Shen +12
Self-improving agents are state-of-the-art on agentic coding benchmarks, yet their search methods assume a stationary evaluation criterion. This ignores a central feature of evolut…
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji +5
Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators. Mixture-of-Experts (MoEs) architectures partially…
PreLort: Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity
Muhammad Waseem, Nurbek Tastan, Andrej Jovanovic +4
Federated fine-tuning of large language models using parameter-efficient methods such as LoRA enables privacy-preserving adaptation of foundation models. Heterogeneous hardware res…
Beyond Scaling: Agents Are Heading to the Edge
Chunlin Tian, Dongqi Cai, Wanru Zhao +1
The bottleneck of useful agentic intelligence has shifted from compressing world knowledge into a single model to executing a coordinated system. This position paper argues that pe…
Response-Conditioned Parallel-to-Sequential Orchestration for Multi-Agent Systems
Nurbek Tastan, Alex Iacob, Lorenzo Sani +4
Multi-agent systems can solve complex tasks through collaboration between multiple Large Language Model agents. Existing collaboration frameworks typically operate in either a para…
EdgeFlowerTune: Evaluating Federated LLM Fine-Tuning Under Realistic Edge System Constraints
Jiaxiang Geng, Yiyi Lu, Lunyu Zhao +3
Federated fine-tuning offers a promising paradigm for adapting large language models (LLMs) on edge devices by leveraging the rich, diverse, and continuously generated data from sm…
Task-Centric Personalized Federated Fine-Tuning of Language Models
Gabriel U. Talasso, Meghdad Kurmanji, Allan M. de Souza +2
Federated Learning (FL) has emerged as a promising technique for training language models on distributed and private datasets of diverse tasks. However, aggregating models trained…
-FUM: Federated Unlearning via min--max and -divergence
Radmehr Karimian, Amirhossein Bagheri, Meghdad Kurmanji +2
Federated learning (FL) enables collaborative training while keeping raw data on clients. Deletion requests and the discovery of poisoned data create a need to remove selected cont…
LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
Andrej Jovanović, Alex Iacob, Mher Safaryan +6
Distributed training of foundation models via is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they…
Computational Compliance for AI Regulation: Blueprint for a New Research Domain
Bill Marino, Nicholas D. Lane
The era of AI regulation (AIR) is upon us. But AI systems, we argue, will not be able to comply with these regulations at the necessary speed and scale by continuing to rely on tra…
MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
Alex Iacob, Andrej Jovanovic, Mher Safaryan +6
Training large models with distributed data parallelism (DDP) requires frequent communication of gradients across workers, which can saturate bandwidth. Infrequent communication st…
AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling
Preslav Aleksandrov, Meghdad Kurmanji, Fernando Garcia Redondo +7
We introduce the Autoregressive Block-Based Iterative Encoder (AbbIE), a novel recursive generalization of the encoder-only Transformer architecture, which achieves better perplexi…
Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages
Wanru Zhao, Yihong Chen, Royson Lee +4
Pre-trained large language models (LLMs) have become a cornerstone of modern natural language processing, with their capabilities extending across a wide range of applications and…
SEAT: Sparse Entity-Aware Tuning for Knowledge Adaptation while Preserving Epistemic Abstention
William F. Shen, Xinchi Qiu, Nicola Cancedda +1
Adapting LLMs with new knowledge is increasingly important, but standard fine-tuning often erodes aligned epistemic abstention: the ability to acknowledge when the model does not k…
DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
Alex Iacob, Lorenzo Sani, Mher Safaryan +8
Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synch…
Editing as Unlearning: Are Knowledge Editing Methods Strong Baselines for Large Language Model Unlearning?
Zexi Li, Xiangzhu Wang, William F. Shen +5
Large language Model (LLM) unlearning, i.e., selectively removing information from LLMs, is vital for responsible model deployment. Differently, LLM knowledge editing aims to modif…
SparsyFed: Sparse Adaptive Federated Training
Adriano Guastella, Lorenzo Sani, Alex Iacob +3
Sparse training is often adopted in cross-device federated learning (FL) environments where constrained devices collaboratively train a machine learning model on private data by ex…
Evidencing Unauthorized Training Data from AI Generated Content using Information Isotopes
Qi Tao, Yin Jinhua, Cai Dongqi +10
In light of scaling laws, many AI institutions are intensifying efforts to construct advanced AIs on extensive collections of high-quality human data. However, in a rush to stay co…
Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis
Jiahao Gai, Hao Mark Chen, Zhican Wang +4
Recent advances in code generation have illuminated the potential of employing large language models (LLMs) for general-purpose programming languages such as Python and C++, openin…