362 citations · 2.5k across the 76 of their papers we have counts for
36 papers · 1 filter
Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidance
Jacob L. Block, Mehryar Mohri, Aryan Mokhtari +1
We study machine unlearning in large generative models by framing the task as density ratio estimation to a target distribution rather than supervised fine-tuning. While classifier…
Machine Unlearning under Overparameterization
Jacob L. Block, Aryan Mokhtari, Sanjay Shakkottai
Machine unlearning algorithms aim to remove the influence of specific training samples, ideally recovering the model that would have resulted from training on the remaining data al…
Provable Meta-Learning with Low-Rank Adaptations
Jacob L. Block, Sundararajan Srinivasan, Liam Collins +2
The power of foundation models (FMs) lies in their capacity to learn highly expressive representations that can be adapted to a broad spectrum of tasks. However, these pretrained m…
In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
Liam Collins, Advait Parulekar, Aryan Mokhtari +2
A striking property of transformers is their ability to perform in-context learning (ICL), a machine learning framework in which the learner is presented with a novel context durin…
Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks
Liam Collins, Hamed Hassani, Mahdi Soltanolkotabi +2
An increasingly popular machine learning paradigm is to pretrain a neural network (NN) on many tasks offline, then adapt it to downstream tasks, often by re-training only the last…
Greedy Pruning with Group Lasso Provably Generalizes for Matrix Sensing
Nived Rajaraman, Devvrit, Aryan Mokhtari +1
Pruning schemes have been widely used in practice to reduce the complexity of trained models with a massive number of parameters. In fact, several practical studies have shown that…
InfoNCE Loss Provably Learns Cluster-Preserving Representations
Advait Parulekar, Liam Collins, Karthikeyan Shanmugam +2
The goal of contrasting learning is to learn a representation that preserves underlying clusters by keeping samples with similar content, e.g. the ``dogness'' of a dog, close to ea…
Network Adaptive Federated Learning: Congestion and Lossy Compression
Parikshit Hegde, Gustavo de Veciana, Aryan Mokhtari
In order to achieve the dual goals of privacy and learning across distributed data, Federated Learning (FL) systems rely on frequent exchanges of large files (model updates) betwee…
Future Gradient Descent for Adapting the Temporal Shifting Data Distribution in Online Recommendation Systems
Mao Ye, Ruichen Jiang, Haoxiang Wang +6
One of the key challenges of learning an online recommendation model is the temporal domain shift, which causes the mismatch between the training and testing data distribution and…
Straggler-Resilient Personalized Federated Learning
Isidoros Tziotis, Zebang Shen, Ramtin Pedarsani +2
Federated Learning is an emerging learning paradigm that allows training models from samples distributed across a large network of clients while respecting privacy and communicatio…
FedAvg with Fine Tuning: Local Updates Lead to Representation Learning
Liam Collins, Hamed Hassani, Aryan Mokhtari +1
The Federated Averaging (FedAvg) algorithm, which consists of alternating between a few local stochastic gradient updates at client nodes, followed by a model averaging update at t…
MAML and ANIL Provably Learn Representations
Liam Collins, Aryan Mokhtari, Sewoong Oh +1
Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an express…
Exploiting Shared Representations for Personalized Federated Learning
Liam Collins, Hamed Hassani, Aryan Mokhtari +1
Deep neural networks have shown the ability to extract universal feature representations from data such as images and text that have been useful for a variety of learning tasks. Ho…
Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Tasks
Alireza Fallah, Aryan Mokhtari, Asuman Ozdaglar
In this paper, we study the generalization properties of Model-Agnostic Meta-Learning (MAML) algorithms for supervised learning problems. We focus on the setting in which we train…
Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity
Amirhossein Reisizadeh, Isidoros Tziotis, Hamed Hassani +2
Federated Learning is a novel paradigm that involves learning from data samples distributed across a large network of clients while the data remains local. It is, however, known th…
How Does the Task Landscape Affect MAML Performance?
Liam Collins, Aryan Mokhtari, Sanjay Shakkottai
Model-Agnostic Meta-Learning (MAML) has become increasingly popular for training models that can quickly adapt to new tasks via one or few stochastic gradient descent steps. Howeve…
Submodular Meta-Learning
Arman Adibi, Aryan Mokhtari, Hamed Hassani
In this paper, we introduce a discrete variant of the meta-learning framework. Meta-learning aims at exploiting prior experience and data to improve performance on future tasks. By…
Federated Learning with Compression: Unified Analysis and Sharp Guarantees
Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari +1
In federated learning, communication cost is often a critical bottleneck to scale up distributed optimization algorithms to collaboratively learn a model from millions of devices w…
Safe Learning under Uncertain Objectives and Constraints
Mohammad Fereydounian, Zebang Shen, Aryan Mokhtari +2
In this paper, we consider non-convex optimization problems under \textit{unknown} yet safety-critical constraints. Such problems naturally arise in a variety of domains including…
Hybrid Model for Anomaly Detection on Call Detail Records by Time Series Forecasting
Aryan Mokhtari, Leyla Sadighi, Behnam Bahrak +1
Mobile network operators store an enormous amount of information like log files that describe various events and users' activities. Analysis of these logs might be used in many cri…
Personalized Federated Learning: A Meta-Learning Approach
Alireza Fallah, Aryan Mokhtari, Asuman Ozdaglar
In Federated Learning, we aim to train models across multiple computing units (users), while users can only communicate with a common central server, without exchanging their data…
On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning
Alireza Fallah, Kristian Georgiev, Aryan Mokhtari +1
We consider Model-Agnostic Meta-Learning (MAML) methods for Reinforcement Learning (RL) problems, where the goal is to find a policy using data from several tasks represented by Ma…
Task-Robust Model-Agnostic Meta-Learning
Liam Collins, Aryan Mokhtari, Sanjay Shakkottai
Meta-learning methods have shown an impressive ability to train models that rapidly learn new tasks. However, these methods only aim to perform well in expectation over tasks comin…
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani +2
Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systems-orient…