Compositional ADAM: An Adaptive Compositional Solver

Abstract

In this paper, we present C-ADAM, the first adaptive solver for compositional problems involving a non-linear functional nesting of expected values. We proof that C-ADAM converges to a stationary point in $\mathcal{O}({\delta }^{-2.25})$ with $\delta$ being a precision parameter. Moreover, we demonstrate the importance of our results by bridging, for the first time, model-agnostic meta-learning (MAML) and compositional optimisation showing fastest known rates for deep network adaptation to-date. Finally, we validate our findings in a set of experiments from portfolio optimisation and meta-learning. Our results manifest significant sample complexity reductions compared to both standard and compositional solvers.

References in corpus (20)

• ADADELTA: An Adaptive Learning Rate Method
• End to End Learning for Self-Driving Cars
• On the Convergence of Adam and Beyond
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Bayesian Model-Agnostic Meta-Learning
• An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedback
• Variance Reduction for Faster Non-Convex Optimization
• Finite-sum Composition Optimization via Variance Reduced Gradient Descent
• Wasserstein Robust Reinforcement Learning
• Accelerating Stochastic Composition Optimization
• Toward Multimodal Model-Agnostic Meta-Learning
• Variance Reduced methods for Non-convex Composition Optimization
• Improved Sample Complexity for Stochastic Compositional Variance Reduced Gradient
• Sparse Additive Models
• Learning to Communicate Implicitly By Actions
• A Viscosity Approach to Stochastic Differential Games of Control and Stopping Involving Impulsive Control
• Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning
• Derivative-Free & Order-Robust Optimisation

Cited by in corpus (7)

• Gryffin: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge
• Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimization
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• Efficient Semi-Implicit Variational Inference