6 citations · 10 across the 13 of their papers we have counts for
6 papers · 1 filter
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6
Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) t…
Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization
Alexander Long, Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan +5
We consider a decentralized setup in which the participants collaboratively train and serve a large neural network, and where each participant only processes a subset of the model.…
NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7
The rapid progress of large language models (LLMs) is increasingly constrained by memory and deployment costs, motivating compression methods for practical deployment. Many state-o…
AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism
Thalaiyasingam Ajanthan, Sameera Ramasinghe, Gil Avraham +5
Data and pipeline parallelism are key strategies for scaling neural network training across distributed devices, but their high communication cost necessitates co-located computing…
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2
Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While…
Nesterov Method for Asynchronous Pipeline Parallel Optimization
Thalaiyasingam Ajanthan, Sameera Ramasinghe, Yan Zuo +2
Pipeline Parallelism (PP) enables large neural network training on small, interconnected devices by splitting the model into multiple stages. To maximize pipeline utilization, asyn…