Optimization Problems for Machine Learning: A Survey

Abstract

This paper surveys the machine learning literature and presents in an optimization framework several commonly used machine learning approaches. Particularly, mathematical optimization models are presented for regression, classification, clustering, deep learning, and adversarial learning, as well as new emerging applications in machine teaching, empirical model learning, and Bayesian network structure learning. Such models can benefit from the advancement of numerical optimization techniques which have already played a distinctive role in several machine learning settings. The strengths and the shortcomings of these models are discussed and potential research directions and open problems are highlighted.

References in corpus (47)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Intriguing properties of neural networks
• Towards A Rigorous Science of Interpretable Machine Learning
• Ensemble Adversarial Training: Attacks and Defenses
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Neural Architecture Search: A Survey
• On the Number of Linear Regions of Deep Neural Networks
• Poisoning Attacks against Support Vector Machines
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Linear Dimensionality Reduction: Survey, Insights, and Generalizations
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Learning Activation Functions to Improve Deep Neural Networks
• Evaluating Robustness of Neural Networks with Mixed Integer Programming
• A Survey on Neural Architecture Search
• Bayesian network learning with cutting planes
• Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso
• A Unified View of Piecewise Linear Neural Network Verification
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• An Overview of Machine Teaching
• A hybrid algorithm for Bayesian network structure learning with application to multi-label learning
• Advances in Bayesian Network Learning using Integer Programming
• Generalized Additive Model Selection
• Data Poisoning Attacks against Online Learning
• Sparsity in Optimal Randomized Classification Trees
• Automated Verification of Neural Networks: Advances, Challenges and Perspectives
• Learning optimization models in the presence of unknown relations
• Optimal randomized classification trees
• Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon
• Sparse principal component regression with adaptive loading
• A general framework for locating hyperplanes to fitting set of points
• Advances in Learning Bayesian Networks of Bounded Treewidth
• Rank-one Convexification for Sparse Regression
• Machine Teaching for Bayesian Learners in the Exponential Family
• The Teaching Dimension of Linear Learners
• Combinatorial Attacks on Binarized Neural Networks
• Activation Ensembles for Deep Neural Networks

Cited by in corpus (14)

• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• A Novel Robust Optimization Model for Nonlinear Support Vector Machine
• An Exact Algorithm for Semi-supervised Minimum Sum-of-Squares Clustering
• Global Optimization: A Machine Learning Approach
• Margin Optimal Classification Trees
• The Backbone Method for Ultra-High Dimensional Sparse Machine Learning
• AutoDOViz: Human-Centered Automation for Decision Optimization
• Global Optimization via Optimal Decision Trees
• Infrequent adverse event prediction in low carbon energy production using machine learning
• An algorithm for clustering with confidence-based must-link and cannot-link constraints
• A Balanced Approach of Rapid Genetic Exploration and Surrogate Exploitation for Hyperparameter Optimization
• Community Detection in the Stochastic Block Model by Mixed Integer Programming
• Training Humans to Train Robots Dynamic Motor Skills
• Fenton-Wilkinson Order Statistics and German Tanks: A Case Study of an Orienteering Relay Race