20 citations · 34 across the 18 of their papers we have counts for
19 papers
Communication-Efficient LLM Adaptation over Decentralized GPU Meshes
Sameera Ramasinghe, Shamane Siriwardhana, Thalaiyasingam Ajanthan +8
Decentralized training enables large-model training over low-end GPUs and internet-grade connections, but communication along both data-parallel and pipeline-parallel axes becomes…
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9
Training large language models at the multi-billion to trillion parameter scale is confined to datacenters, where data-parallel (DP) and model-parallel (MP) techniques presume homo…
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7
To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently,…
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6
Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) t…
Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training
Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +5
Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks,…
Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization
Alexander Long, Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan +5
We consider a decentralized setup in which the participants collaboratively train and serve a large neural network, and where each participant only processes a subset of the model.…
NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7
The rapid progress of large language models (LLMs) is increasingly constrained by memory and deployment costs, motivating compression methods for practical deployment. Many state-o…
SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training
Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +5
Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes. While existing Byzantine-tolerant literature addresses d…
AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism
Thalaiyasingam Ajanthan, Sameera Ramasinghe, Gil Avraham +5
Data and pipeline parallelism are key strategies for scaling neural network training across distributed devices, but their high communication cost necessitates co-located computing…
Exploiting Inter-Sample Information for Long-tailed Out-of-Distribution Detection
Nimeshika Udayangani, Hadi M. Dolatabadi, Sarah Erfani +1
Detecting out-of-distribution (OOD) data is essential for safe deployment of deep neural networks (DNNs). This problem becomes particularly challenging in the presence of long-tail…
Round Trip Translation Defence against Large Language Model Jailbreaking Attacks
Canaan Yung, Hadi Mohaghegh Dolatabadi, Sarah Erfani +1
Large language models (LLMs) are susceptible to social-engineered attacks that are human-interpretable but require a high level of comprehension for LLMs to counteract. Existing de…
Be Persistent: Towards a Unified Solution for Mitigating Shortcuts in Deep Learning
Hadi M. Dolatabadi, Sarah M. Erfani, Christopher Leckie
Deep neural networks (DNNs) are vulnerable to shortcut learning: rather than learning the intended task, they tend to draw inconclusive relationships between their inputs and outpu…
The Devil's Advocate: Shattering the Illusion of Unexploitable Data using Diffusion Models
Hadi M. Dolatabadi, Sarah Erfani, Christopher Leckie
Protecting personal data against exploitation of machine learning models is crucial. Recently, availability attacks have shown great promise to provide an extra layer of protection…
COLLIDER: A Robust Training Framework for Backdoor Data
Hadi M. Dolatabadi, Sarah Erfani, Christopher Leckie
Deep neural network (DNN) classifiers are vulnerable to backdoor attacks. An adversary poisons some of the training data in such attacks by installing a trigger. The goal is to mak…
Adversarial Coreset Selection for Efficient Robust Training
Hadi M. Dolatabadi, Sarah Erfani, Christopher Leckie
Neural networks are vulnerable to adversarial attacks: adding well-crafted, imperceptible perturbations to their input can modify their output. Adversarial training is one of the m…
-Robustness and Beyond: Unleashing Efficient Adversarial Training
Hadi M. Dolatabadi, Sarah Erfani, Christopher Leckie
Neural networks are vulnerable to adversarial attacks: adding well-crafted, imperceptible perturbations to their input can modify their output. Adversarial training is one of the m…
Black-box Adversarial Example Generation with Normalizing Flows
Hadi M. Dolatabadi, Sarah Erfani, Christopher Leckie
Deep neural network classifiers suffer from adversarial vulnerability: well-crafted, unnoticeable changes to the input data can affect the classifier decision. In this regard, the…
AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows
Hadi M. Dolatabadi, Sarah Erfani, Christopher Leckie
Deep learning classifiers are susceptible to well-crafted, imperceptible variations of their inputs, known as adversarial attacks. In this regard, the study of powerful attack mode…
Invertible Generative Modeling using Linear Rational Splines
Hadi M. Dolatabadi, Sarah Erfani, Christopher Leckie
Normalizing flows attempt to model an arbitrary probability distribution through a set of invertible mappings. These transformations are required to achieve a tractable Jacobian de…