Exploiting Smoothness in Statistical Learning, Sequential Prediction, and Stochastic Optimization

Abstract

In the last several years, the intimate connection between convex optimization and learning problems, in both statistical and sequential frameworks, has shifted the focus of algorithmic machine learning to examine this interplay. In particular, on one hand, this intertwinement brings forward new challenges in reassessment of the performance of learning algorithms including generalization and regret bounds under the assumptions imposed by convexity such as analytical properties of loss functions (e.g., Lipschitzness, strong convexity, and smoothness). On the other hand, emergence of datasets of an unprecedented size, demands the development of novel and more efficient optimization algorithms to tackle large-scale learning problems. The overarching goal of this thesis is to reassess the smoothness of loss functions in statistical learning, sequential prediction/online learning, and stochastic optimization and explicate its consequences. In particular we examine how smoothness of loss function could be beneficial or detrimental in these settings in terms of sample complexity, statistical consistency, regret analysis, and convergence rate, and investigate how smoothness can be leveraged to devise more efficient learning algorithms.

Ph.D. Thesis

References in corpus (8)

• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• Better Mini-Batch Algorithms via Accelerated Gradient Methods
• Projection-free Online Learning
• Minimizing The Misclassification Error Rate Using a Surrogate Convex Loss
• Learning From An Optimization Viewpoint
• A Smoothing Stochastic Gradient Method for Composite Optimization
• O(logT) Projections for Stochastic Optimization of Smooth and Strongly Convex Functions