362 citations · 2.5k across the 76 of their papers we have counts for
83 papers
Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidance
Jacob L. Block, Mehryar Mohri, Aryan Mokhtari +1
We study machine unlearning in large generative models by framing the task as density ratio estimation to a target distribution rather than supervised fine-tuning. While classifier…
Machine Unlearning under Overparameterization
Jacob L. Block, Aryan Mokhtari, Sanjay Shakkottai
Machine unlearning algorithms aim to remove the influence of specific training samples, ideally recovering the model that would have resulted from training on the remaining data al…
Provable Meta-Learning with Low-Rank Adaptations
Jacob L. Block, Sundararajan Srinivasan, Liam Collins +2
The power of foundation models (FMs) lies in their capacity to learn highly expressive representations that can be adapted to a broad spectrum of tasks. However, these pretrained m…
In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
Liam Collins, Advait Parulekar, Aryan Mokhtari +2
A striking property of transformers is their ability to perform in-context learning (ICL), a machine learning framework in which the learner is presented with a novel context durin…
Projection-Free Methods for Stochastic Simple Bilevel Optimization with Convex Lower-level Problem
Jincheng Cao, Ruichen Jiang, Nazanin Abolfazli +2
In this paper, we study a class of stochastic bilevel optimization problems, also known as stochastic simple bilevel optimization, where we minimize a smooth stochastic objective f…
Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks
Liam Collins, Hamed Hassani, Mahdi Soltanolkotabi +2
An increasingly popular machine learning paradigm is to pretrain a neural network (NN) on many tasks offline, then adapt it to downstream tasks, often by re-training only the last…
Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization
Ruichen Jiang, Aryan Mokhtari
In this paper, we propose an accelerated quasi-Newton proximal extragradient (A-QPNE) method for solving unconstrained smooth convex optimization problems. With access only to the…
Limited-Memory Greedy Quasi-Newton Method with Non-asymptotic Superlinear Convergence Rate
Zhan Gao, Aryan Mokhtari, Alec Koppel
Non-asymptotic convergence analysis of quasi-Newton methods has gained attention with a landmark result establishing an explicit local superlinear rate of O. The…
An Inexact Conditional Gradient Method for Constrained Bilevel Optimization
Nazanin Abolfazli, Ruichen Jiang, Aryan Mokhtari +1
Bilevel optimization is an important class of optimization problems where one optimization problem is nested within another. While various methods have emerged to address unconstra…
Greedy Pruning with Group Lasso Provably Generalizes for Matrix Sensing
Nived Rajaraman, Devvrit, Aryan Mokhtari +1
Pruning schemes have been widely used in practice to reduce the complexity of trained models with a massive number of parameters. In fact, several practical studies have shown that…
InfoNCE Loss Provably Learns Cluster-Preserving Representations
Advait Parulekar, Liam Collins, Karthikeyan Shanmugam +2
The goal of contrasting learning is to learn a representation that preserves underlying clusters by keeping samples with similar content, e.g. the ``dogness'' of a dog, close to ea…
Online Learning Guided Curvature Approximation: A Quasi-Newton Method with Global Non-Asymptotic Superlinear Convergence
Ruichen Jiang, Qiujiang Jin, Aryan Mokhtari
Quasi-Newton algorithms are among the most popular iterative methods for solving unconstrained minimization problems, largely due to their favorable superlinear convergence propert…
Network Adaptive Federated Learning: Congestion and Lossy Compression
Parikshit Hegde, Gustavo de Veciana, Aryan Mokhtari
In order to achieve the dual goals of privacy and learning across distributed data, Federated Learning (FL) systems rely on frequent exchanges of large files (model updates) betwee…
Conditional Gradient Methods
Gábor Braun, Alejandro Carderera, Cyrille W. Combettes +4
The purpose of this survey is to serve both as a gentle introduction and a coherent overview of state-of-the-art Frank--Wolfe algorithms, also called conditional gradient algorithm…
Future Gradient Descent for Adapting the Temporal Shifting Data Distribution in Online Recommendation Systems
Mao Ye, Ruichen Jiang, Haoxiang Wang +6
One of the key challenges of learning an online recommendation model is the temporal domain shift, which causes the mismatch between the training and testing data distribution and…
Straggler-Resilient Personalized Federated Learning
Isidoros Tziotis, Zebang Shen, Ramtin Pedarsani +2
Federated Learning is an emerging learning paradigm that allows training models from samples distributed across a large network of clients while respecting privacy and communicatio…
A Conditional Gradient-based Method for Simple Bilevel Optimization with Convex Lower-level Problem
Ruichen Jiang, Nazanin Abolfazli, Aryan Mokhtari +1
In this paper, we study a class of bilevel optimization problems, also known as simple bilevel optimization, where we minimize a smooth objective function over the optimal solution…
Statistical and Computational Complexities of BFGS Quasi-Newton Method for Generalized Linear Models
Qiujiang Jin, Tongzheng Ren, Nhat Ho +1
The gradient descent (GD) method has been used widely to solve parameter estimation in generalized linear models (GLMs), a generalization of linear models when the link function ca…
FedAvg with Fine Tuning: Local Updates Lead to Representation Learning
Liam Collins, Hamed Hassani, Aryan Mokhtari +1
The Federated Averaging (FedAvg) algorithm, which consists of alternating between a few local stochastic gradient updates at client nodes, followed by a model averaging update at t…
Provably Private Distributed Averaging Consensus: An Information-Theoretic Approach
Mohammad Fereydounian, Aryan Mokhtari, Ramtin Pedarsani +1
In this work, we focus on solving a decentralized consensus problem in a private manner. Specifically, we consider a setting in which a group of nodes, connected through a network,…
Sharpened Quasi-Newton Methods: Faster Superlinear Rate and Larger Local Convergence Neighborhood
Qiujiang Jin, Alec Koppel, Ketan Rajawat +1
Non-asymptotic analysis of quasi-Newton methods have gained traction recently. In particular, several works have established a non-asymptotic superlinear rate of $\mathcal{O}((1/\s…
Generalized Optimistic Methods for Convex-Concave Saddle Point Problems
Ruichen Jiang, Aryan Mokhtari
The optimistic gradient method has seen increasing popularity for solving convex-concave saddle point problems. To analyze its iteration complexity, a recent work [arXiv:1906.01115…
The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance
Matthew Faw, Isidoros Tziotis, Constantine Caramanis +3
We study convergence rates of AdaGrad-Norm as an exemplar of adaptive stochastic gradient methods (SGD), where the step sizes change based on observed stochastic gradients, for min…
MAML and ANIL Provably Learn Representations
Liam Collins, Aryan Mokhtari, Sewoong Oh +1
Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an express…