3k citations · 4.4k across the 29 of their papers we have counts for
31 papers
Correlated quantization for distributed mean estimation and optimization
Ananda Theertha Suresh, Ziteng Sun, Jae Hun Ro +1
We study the problem of distributed mean estimation and optimization under communication constraints. We propose a correlated quantization protocol whose leading term in the error…
FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients
Jianyu Wang, Hang Qi, Ankit Singh Rawat +4
In classical federated learning, the clients contribute to the overall training by communicating local updates for the underlying model on their private data to a coordinating serv…
HD-cos Networks: Efficient Neural Architectures for Secure Multi-Party Computation
Wittawat Jitkrittum, Michal Lukasik, Ananda Theertha Suresh +2
Multi-party computation (MPC) is a branch of cryptography where multiple non-colluding parties execute a well designed protocol to securely compute a function. With the non-colludi…
A Field Guide to Federated Optimization
Jianyu Wang, Zachary Charles, Zheng Xu +50
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy prote…
Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
Ankit Singh Rawat, Aditya Krishna Menon, Wittawat Jitkrittum +4
Negative sampling schemes enable efficient training given a large number of classes, by offering a means to approximate a computationally expensive loss function that takes all lab…
Modifying Memories in Transformer Models
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer +4
Large Transformer models have achieved impressive performance in many natural language tasks. In particular, Transformer based language models have been shown to have great capabil…
Semantic Label Smoothing for Sequence to Sequence Problems
Michal Lukasik, Himanshu Jain, Aditya Krishna Menon +4
Label smoothing has been shown to be an effective regularization strategy in classification, that prevents overfitting and helps in label de-noising. However, extending such method…
Learning discrete distributions: user vs item-level privacy
Yuhan Liu, Ananda Theertha Suresh, Felix Yu +2
Much of the literature on differential privacy focuses on item-level privacy, where loosely speaking, the goal is to provide privacy per item or training example. However, recently…
Self-supervised Learning for Large-scale Item Recommendations
Tiansheng Yao, Xinyang Yi, Derek Zhiyuan Cheng +8
Large scale recommender models find most relevant items from huge catalogs, and they play a critical role in modern search and recommendation systems. To model the input space with…
Doubly-stochastic mining for heterogeneous retrieval
Ankit Singh Rawat, Aditya Krishna Menon, Andreas Veit +3
Modern retrieval problems are characterised by training sets with potentially billions of labels, and heterogeneous data distributions across subpopulations (e.g., users of a retri…
Federated Learning with Only Positive Labels
Felix X. Yu, Ankit Singh Rawat, Aditya Krishna Menon +1
We consider learning a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a single class. As a result,…
Pre-training Tasks for Embedding-based Large-scale Retrieval
Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang +2
We consider the large-scale query-document retrieval problem: given a query (e.g., a question), return the set of relevant documents (e.g., paragraphs containing the answer) from a…
Advances and Open Problems in Federated Learning
Peter Kairouz, H. Brendan McMahan, Brendan Avent +56
Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole organizations) collaboratively train a model under the orchestration of a cen…
AdaCliP: Adaptive Clipping for Private SGD
Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X. Yu +2
Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic…
Sampled Softmax with Random Fourier Features
Ankit Singh Rawat, Jiecao Chen, Felix Yu +2
The computational cost of training with softmax cross entropy loss grows linearly with the number of classes. For the settings where a large number of classes are involved, a commo…
Stochastic Negative Mining for Learning with Large Output Spaces
Sashank J. Reddi, Satyen Kale, Felix Yu +3
We consider the problem of retrieving the most relevant labels for a given input when the size of the output space is very large. Retrieval methods are modeled as set-valued classi…
Heated-Up Softmax Embedding
Xu Zhang, Felix Xinnan Yu, Svebor Karaman +2
Metric learning aims at learning a distance which is consistent with the semantic meaning of the samples. The problem is generally solved by learning an embedding for each sample s…
Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling
Shanshan Wu, Alexandros G. Dimakis, Sujay Sanghavi +5
Linear encoding of sparse vectors is widely popular, but is commonly data-independent -- missing any possible extra (but a priori unknown) structure beyond sparsity. In this paper…
cpSGD: Communication-efficient and differentially-private distributed SGD
Naman Agarwal, Ananda Theertha Suresh, Felix Yu +2
Distributed stochastic gradient descent is an important subroutine in distributed learning. A setting of particular interest is when the clients are mobile devices, where two impor…
Lattice Rescoring Strategies for Long Short Term Memory Language Models in Speech Recognition
Shankar Kumar, Michael Nirschl, Daniel Holtmann-Rice +3
Recurrent neural network (RNN) language models (LMs) and Long Short Term Memory (LSTM) LMs, a variant of RNN LMs, have been shown to outperform traditional N-gram LMs on speech rec…
Learning Spread-out Local Feature Descriptors
Xu Zhang, Felix X. Yu, Sanjiv Kumar +1
We propose a simple, yet powerful regularization technique that can be used to significantly improve both the pairwise and triplet losses in learning local feature descriptors. The…
Distributed Mean Estimation with Limited Communication
Ananda Theertha Suresh, Felix X. Yu, Sanjiv Kumar +1
Motivated by the need for distributed learning and optimization algorithms with low communication cost, we study communication efficient algorithms for distributed mean estimation.…
Orthogonal Random Features
Felix X. Yu, Ananda Theertha Suresh, Krzysztof Choromanski +2
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal…
Federated Learning: Strategies for Improving Communication Efficiency
Jakub Konečný, H. Brendan McMahan, Felix X. Yu +3
Federated Learning is a machine learning setting where the goal is to train a high-quality centralized model while training data remains distributed over a large number of clients…