1.1k citations · 2.1k across the 32 of their papers we have counts for
40 papers
Communication-Efficient LLM Adaptation over Decentralized GPU Meshes
Sameera Ramasinghe, Shamane Siriwardhana, Thalaiyasingam Ajanthan +8
Decentralized training enables large-model training over low-end GPUs and internet-grade connections, but communication along both data-parallel and pipeline-parallel axes becomes…
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9
Training large language models at the multi-billion to trillion parameter scale is confined to datacenters, where data-parallel (DP) and model-parallel (MP) techniques presume homo…
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7
To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently,…
Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training
Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +5
Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks,…
Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization
Alexander Long, Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan +5
We consider a decentralized setup in which the participants collaboratively train and serve a large neural network, and where each participant only processes a subset of the model.…
NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7
The rapid progress of large language models (LLMs) is increasingly constrained by memory and deployment costs, motivating compression methods for practical deployment. Many state-o…
SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training
Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +5
Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes. While existing Byzantine-tolerant literature addresses d…
AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism
Thalaiyasingam Ajanthan, Sameera Ramasinghe, Gil Avraham +5
Data and pipeline parallelism are key strategies for scaling neural network training across distributed devices, but their high communication cost necessitates co-located computing…
Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappings
Evan Markou, Thalaiyasingam Ajanthan, Stephen Gould
Many high-dimensional optimisation problems exhibit rich geometric structures in their set of minimisers, often forming smooth manifolds due to over-parametrisation or symmetries.…
Nesterov Method for Asynchronous Pipeline Parallel Optimization
Thalaiyasingam Ajanthan, Sameera Ramasinghe, Yan Zuo +2
Pipeline Parallelism (PP) enables large neural network training on small, interconnected devices by splitting the model into multiple stages. To maximize pipeline utilization, asyn…
Scaling Prompt Instructed Zero Shot Composed Image Retrieval with Image-Only Data
Yiqun Duan, Sameera Ramasinghe, Stephen Gould +1
Composed Image Retrieval (CIR) is the task of retrieving images matching a reference image augmented with a text, where the text describes changes to the reference image in natural…
Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame
Evan Markou, Thalaiyasingam Ajanthan, Stephen Gould
Neural Collapse (NC) is a recently observed phenomenon in neural networks that characterises the solution space of the final classifier layer when trained until zero training loss.…
Learning Visual Hierarchies in Hyperbolic Space for Image Retrieval
Ziwei Wang, Sameera Ramasinghe, Chenchen Xu +3
Structuring latent representations in a hierarchical manner enables models to learn patterns at multiple levels of abstraction. However, most prevalent image understanding models f…
Self-Supervision Improves Diffusion Models for Tabular Data Imputation
Yixin Liu, Thalaiyasingam Ajanthan, Hisham Husain +1
The ubiquity of missing data has sparked considerable attention and focus on tabular data imputation methods. Diffusion models, recognized as the cutting-edge technique for data ge…
Adaptive Cross Batch Normalization for Metric Learning
Thalaiyasingam Ajanthan, Matt Ma, Anton van den Hengel +1
Metric learning is a fundamental problem in computer vision whereby a model is trained to learn a semantically useful embedding space via ranking losses. Traditionally, the effecti…
Understanding and Improving the Role of Projection Head in Self-Supervised Learning
Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel +1
Self-supervised learning (SSL) aims to produce useful feature representations without access to any human-labeled data annotations. Due to the success of recent SSL methods based o…
Retrieval Augmented Classification for Long-Tail Visual Recognition
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan +6
We introduce Retrieval Augmented Classification (RAC), a generic approach to augmenting standard image classification pipelines with an explicit retrieval module. RAC consists of a…
Few-shot Weakly-Supervised Object Detection via Directional Statistics
Amirreza Shaban, Amir Rahimi, Thalaiyasingam Ajanthan +2
Detecting novel objects from few examples has become an emerging topic in computer vision recently. However, these methods need fully annotated training images to learn new object…
RANP: Resource Aware Neuron Pruning at Initialization for 3D CNNs
Zhiwei Xu, Thalaiyasingam Ajanthan, Vibhav Vineet +1
Although 3D Convolutional Neural Networks are essential for most learning based applications involving dense 3D data, their applicability is limited due to excessive memory and com…
Refining Semantic Segmentation with Superpixel by Transparent Initialization and Sparse Encoder
Zhiwei Xu, Thalaiyasingam Ajanthan, Richard Hartley
Although deep learning greatly improves the performance of semantic segmentation, its success mainly lies in object central areas without accurate edges. As superpixels are a popul…
RANP: Resource Aware Neuron Pruning at Initialization for 3D CNNs
Zhiwei Xu, Thalaiyasingam Ajanthan, Vibhav Vineet +1
Although 3D Convolutional Neural Networks (CNNs) are essential for most learning based applications involving dense 3D data, their applicability is limited due to excessive memory…
Post-hoc Calibration of Neural Networks by g-Layers
Amir Rahimi, Thomas Mensink, Kartik Gupta +3
Calibration of neural networks is a critical aspect to consider when incorporating machine learning models in real-world decision-making systems where the confidence of decisions a…
Calibration of Neural Networks using Splines
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan +3
Calibrating neural networks is of utmost importance when employing them in safety-critical applications where the downstream decision making depends on the predicted probabilities.…
Bidirectionally Self-Normalizing Neural Networks
Yao Lu, Stephen Gould, Thalaiyasingam Ajanthan
The problem of vanishing and exploding gradients has been a long-standing obstacle that hinders the effective training of neural networks. Despite various tricks and techniques tha…