204 citations · 476 across the 32 of their papers we have counts for
25 papers · 1 filter
Dynamic estimation of slowly varying sequences
Prashant Gokhale, Mikhail Khodak, Sandeep Silwal
We consider the problem of sequentially approximating functions of each element in a slowly-varying sequence, i.e. one where the magnitude of the difference between the eleme…
Breakeven complexity: A new perspective on neural partial differential equation solvers
Yijing Zhang, Nicholas Roberts, Tanya Marwah +1
Neural surrogate solvers of partial differential equations (PDEs) promise dramatic speedups over numerical methods, especially in scenarios requiring many solves. However, current…
Pre-Generating Multi-Difficulty PDE Data for Few-Shot Neural PDE Solvers
Naman Choudhary, Vedant Singh, Ameet Talwalkar +3
A key aspect of learned partial differential equation (PDE) solvers is that the main cost often comes from generating training data with classical solvers rather than learning the…
Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
Yihe Dong, Lorenzo Noci, Mikhail Khodak +1
The transformer architecture is central to the success of modern Large Language Models (LLMs), in part due to its surprising ability to perform a wide range of tasks - including ma…
SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation
Mikhail Khodak, Lester Mackey, Alexandra Chouldechova +1
Disaggregated evaluation -- estimation of performance of a machine learning model on different subpopulations -- is a core task when assessing performance and group-fairness of AI…
Specialized Foundation Models Struggle to Beat Supervised Baselines
Zongzhe Xu, Ritvik Gupta, Wenduo Cheng +4
Following its success for vision and text, the "foundation model" (FM) paradigm -- pretraining large models on massive data, then fine-tuning on target tasks -- has rapidly expande…
Learning to Relax: Setting Solver Parameters Across a Sequence of Linear System Instances
Mikhail Khodak, Edmond Chow, Maria-Florina Balcan +1
Solving a linear system is a fundamental scientific computing primitive for which numerous solvers and preconditioners have been developed. These come with parameters whose…
Meta-Learning Adversarial Bandit Algorithms
Mikhail Khodak, Ilya Osadchiy, Keegan Harris +4
We study online meta-learning with bandit feedback, with the goal of improving performance across multiple tasks if they are similar according to some natural similarity measure. A…
Cross-Modal Fine-Tuning: Align then Refine
Junhong Shen, Liam Li, Lucio M. Dery +4
Fine-tuning large-scale pretrained models has led to tremendous progress in well-studied modalities such as vision and NLP. However, similar gains have not been observed in many ot…
On Noisy Evaluation in Federated Hyperparameter Tuning
Kevin Kuo, Pratiksha Thaker, Mikhail Khodak +4
Hyperparameter tuning is critical to the success of federated learning applications. Unfortunately, appropriately selecting hyperparameters is challenging in federated networks. Is…
Provably tuning the ElasticNet across instances
Maria-Florina Balcan, Mikhail Khodak, Dravyansh Sharma +1
An important unresolved challenge in the theory of regularization is to set the regularization coefficients of popular techniques like the ElasticNet with general provable guarante…
Meta-Learning Adversarial Bandits
Maria-Florina Balcan, Keegan Harris, Mikhail Khodak +1
We study online learning with bandit feedback across multiple tasks, with the goal of improving average performance across tasks if they are similar according to some natural task-…
AANG: Automating Auxiliary Learning
Lucio M. Dery, Paul Michel, Mikhail Khodak +2
Auxiliary objectives, supplementary learning signals that are introduced to help aid learning on data-starved or highly complex end-tasks, are commonplace in machine learning. Whil…
Efficient Architecture Search for Diverse Tasks
Junhong Shen, Mikhail Khodak, Ameet Talwalkar
While neural architecture search (NAS) has enabled automated machine learning (AutoML) for well-researched areas, its application to tasks beyond computer vision is still under-exp…
Learning Predictions for Algorithms with Predictions
Mikhail Khodak, Maria-Florina Balcan, Ameet Talwalkar +1
A burgeoning paradigm in algorithm design is the field of algorithms with predictions, in which algorithms can take advantage of a possibly-imperfect prediction of some aspect of t…
Learning-to-learn non-convex piecewise-Lipschitz functions
Maria-Florina Balcan, Mikhail Khodak, Dravyansh Sharma +1
We analyze the meta-learning of the initialization and step-size of learning algorithms for piecewise-Lipschitz functions, a non-convex setting with applications to both machine le…
Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
Mikhail Khodak, Renbo Tu, Tian Li +4
Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline. Hyperparameter optimization is even more challenging in federated learning, where models are…
Rethinking Neural Operations for Diverse Tasks
Nicholas Roberts, Mikhail Khodak, Tri Dao +3
An important goal of AutoML is to automate-away the design of neural networks on new tasks in under-explored domains. Motivated by this goal, we study the problem of enabling users…
Geometry-Aware Gradient Algorithms for Neural Architecture Search
Liam Li, Mikhail Khodak, Maria-Florina Balcan +1
Recent state-of-the-art methods for neural architecture search (NAS) exploit gradient-based optimization by relaxing the problem into continuous optimization over architectures and…
A Sample Complexity Separation between Non-Convex and Convex Meta-Learning
Nikunj Saunshi, Yi Zhang, Mikhail Khodak +1
One popular trend in meta-learning is to learn from many training tasks a common initialization for a gradient-based method that can be used to solve a new task with few samples. T…
Advances and Open Problems in Federated Learning
Peter Kairouz, H. Brendan McMahan, Brendan Avent +56
Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole organizations) collaboratively train a model under the orchestration of a cen…
Differentially Private Meta-Learning
Jeffrey Li, Mikhail Khodak, Sebastian Caldas +1
Parameter-transfer is a well-known and versatile approach for meta-learning, with applications including few-shot learning, federated learning, and reinforcement learning. However,…
Adaptive Gradient-Based Meta-Learning Methods
Mikhail Khodak, Maria-Florina Balcan, Ameet Talwalkar
We build a theoretical framework for designing and understanding practical meta-learning methods that integrates sophisticated formalizations of task-similarity with the extensive…
A Theoretical Analysis of Contrastive Unsupervised Representation Learning
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak +2
Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods a…