Approximate Frank-Wolfe Algorithms over Graph-structured Support Sets

Abstract

In this paper, we consider approximate Frank-Wolfe (FW) algorithms to solve convex optimization problems over graph-structured support sets where the linear minimization oracle (LMO) cannot be efficiently obtained in general. We first demonstrate that two popular approximation assumptions (additive and multiplicative gap errors) are not applicable in that no cheap gap-approximate LMO oracle exists. Thus, approximate dual maximization oracles (DMO) are proposed, which approximate the inner product rather than the gap. We prove that the standard FW method using a $\delta$-approximate DMO converges as $\mathcal{O}((1-\delta )\sqrt{s}/\delta )$ in the worst case, and as $\mathcal{O}(L/({\delta }^{2}t))$ over a $\delta$-relaxation of the constraint set. Furthermore, when the solution is on the boundary, a variant of FW converges as $\mathcal{O}(1/t^2)$ under the quadratic growth assumption. Our empirical results suggest that even these improved bounds are pessimistic, showing fast convergence in recovering real-world images with graph-structured sparsity.

35 pages, 11 figures

References in corpus (12)

• Network Lasso: Clustering and Optimization in Large Graphs
• Variance-Reduced and Projection-Free Stochastic Optimization
• Structured sparsity-inducing norms through submodular functions
• Frank-Wolfe Bayesian Quadrature: Probabilistic Integration with Theoretical Guarantees
• Linear-memory and Decomposition-invariant Linearly Convergent Conditional Gradient Algorithm for Structured Polytopes
• Tight convex relaxations for sparse matrix factorization
• A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe
• Barrier Frank-Wolfe for Marginal Inference
• Combinatorial Penalties: Which structures are preserved by convex relaxations?
• Safe Screening for the Generalized Conditional Gradient Method
• On Matching Pursuit and Coordinate Descent
• Blended Matching Pursuit