1.1k citations · 1.3k across the 20 of their papers we have counts for
9 papers · 1 filter
lo-fi: distributed fine-tuning without communication
Mitchell Wortsman, Suchin Gururangan, Shen Li +4
When fine-tuning large neural networks, it is common to use multiple nodes and to communicate gradients at each optimization step. By contrast, we investigate completely local fine…
Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
John Nguyen, Jianyu Wang, Kshitiz Malik +2
An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very differe…
The Hidden Uniform Cluster Prior in Self-Supervised Learning
Mahmoud Assran, Randall Balestriero, Quentin Duval +6
A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e.g., SimCLR, VICReg, SwAV, MSN). We show tha…
Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
John Nguyen, Jianyu Wang, Kshitiz Malik +2
An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very differe…
Positive Unlabeled Contrastive Learning
Anish Acharya, Sujay Sanghavi, Li Jing +4
Self-supervised pretraining on unlabeled data followed by supervised fine-tuning on labeled data is a popular paradigm for learning from limited labeled examples. We extend this pa…
Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity
Kiwan Maeng, Haiyu Lu, Luca Melis +3
Federated learning (FL) is an effective mechanism for data privacy in recommender systems by running machine learning model training on-device. While prior FL optimizations tackled…