71 citations · 253 across the 31 of their papers we have counts for
12 papers · 1 filter
Measuring and Reducing Cross-Vendor Mismatch in Language Models
Erland Hilman Fuadi, Chong Tian, Xiaosong Ma +1
Running the same language model on different graphics processing unit (GPU) vendors can produce different logits, even when the model weights and inputs are the same. We analyze cr…
How Does Controllability Emerge In Language Models During Pretraining?
Jianshu She, Xinyue Li, Eric Xing +2
Language models can be steered by modifying their internal representations to control concepts such as emotion, style, or truthfulness in generation. However, the conditions for an…
Reducing Hyperparameter Tuning Costs in ML, Vision and Language Model Training Pipelines via Memoization-Awareness
Abdelmajid Essofi, Ridwan Salahuddeen, Munachiso Nwadike +5
The training or fine-tuning of machine learning, vision, and language models is often implemented as a pipeline: a sequence of stages encompassing data preparation, model training…
Continual Learning of Nonlinear Independent Representations
Boyang Sun, Ignavier Ng, Guangyi Chen +3
Identifying the causal relations between interested variables plays a pivotal role in representation learning as it provides deep insights into the dataset. Identifiability, as the…
FedNAR: Federated Optimization with Normalized Annealing Regularization
Junbo Li, Ang Li, Chong Tian +3
Weight decay is a standard technique to improve generalization performance in modern deep neural network optimization, and is also widely adopted in federated learning (FL) to prev…
On Optimizing the Communication of Model Parallelism
Yonghao Zhuang, Hexu Zhao, Lianmin Zheng +6
We study a novel and important communication pattern in large-scale model-parallel deep learning (DL), which we call cross-mesh resharding. This pattern emerges when the two paradi…
Cavs: A Vertex-centric Programming Interface for Dynamic Neural Networks
Hao Zhang, Shizhen Xu, Graham Neubig +4
Recent deep learning (DL) models have moved beyond static network architectures to dynamic ones, handling data where the network structure changes every example, such as sequences…
Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
Hao Zhang, Zeyu Zheng, Shizhen Xu +7
Deep learning models can take weeks to train on a single GPU-equipped machine, necessitating scaling out DL training to a GPU-cluster. However, current distributed DL implementatio…
Distributed Multi-Task Relationship Learning
Sulin Liu, Sinno Jialin Pan, Qirong Ho
Multi-task learning aims to learn multiple tasks jointly by exploiting their relatedness to improve the generalization performance for each task. Traditionally, to perform multi-ta…
Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
Hao Zhang, Zhiting Hu, Jinliang Wei +4
Deep learning (DL) has achieved notable successes in many machine learning tasks. A number of frameworks have been developed to expedite the process of designing and training deep…
Distributed Machine Learning via Sufficient Factor Broadcasting
Pengtao Xie, Jin Kyu Kim, Yi Zhou +4
Matrix-parametrized models, including multiclass logistic regression and sparse coding, are used in machine learning (ML) applications ranging from computer vision to computational…
High-Performance Distributed ML at Scale through Parameter Server Consistency Models
Wei Dai, Abhimanu Kumar, Jinliang Wei +3
As Machine Learning (ML) applications increase in data size and model complexity, practitioners turn to distributed clusters to satisfy the increased computational and memory deman…