Overcoming The Limitations of Phase Transition by Higher Order Analysis of Regularization Techniques

Abstract

We study the problem of estimating $\beta \in {\mathbb{R}}^p$ from its noisy linear observations $y=X\beta +w$, where $w\sim N(0,{\sigma }_w^2{I}_{n\times n})$, under the following high-dimensional asymptotic regime: given a fixed number $\delta$, $p\to \infty$, while $n/p\to \delta$. We consider the popular class of ${\ell }_q$-regularized least squares (LQLS) estimators, a.k.a. bridge, given by the optimization problem: \begin{equation*} \hatβ (λ, q ) \in \arg\min_β\frac{1}{2} \|y-Xβ\|_2^2+ λ\|β\|_q^q, \end{equation*} and characterize the almost sure limit of $\frac{1}{p}\Vert \hat{\beta }(\lambda ,q)-\beta {\Vert }_2^2$. The expression we derive for this limit does not have explicit forms and hence are not useful in comparing different algorithms, or providing information in evaluating the effect of $\delta$ or sparsity level of $\beta$. To simplify the expressions, researchers have considered the ideal "no-noise" regime and have characterized the values of $\delta$ for which the almost sure limit is zero. This is known as the phase transition analysis. In this paper, we first perform the phase transition analysis of LQLS. Our results reveal some of the limitations and misleading features of the phase transition analysis. To overcome these limitations, we propose the study of these algorithms under the low noise regime. Our new analysis framework not only sheds light on the results of the phase transition analysis, but also makes an accurate comparison of different regularizers possible.

References in corpus (7)

• Asymptotic properties of bridge estimators in sparse high-dimensional regression models
• Various thresholds for ${\ell }_1$-optimization in compressed sensing
• Block-length dependent thresholds in block-sparse compressed sensing
• False Discoveries Occur Early on the Lasso Path
• Precise Error Analysis of Regularized M-estimators in High-dimensions
• Does ${\ell }_p$-minimization outperform ${\ell }_1$-minimization?
• Robustness in sparse linear models: relative efficiency based on robust approximate message passing

Cited by in corpus (4)

• Does ${\ell }_p$-minimization outperform ${\ell }_1$-minimization?
• Approximate Leave-One-Out for High-Dimensional Non-Differentiable Learning Problems
• Spectral Method for Phase Retrieval: an Expectation Propagation Perspective
• Low noise sensitivity analysis of Lq-minimization in oversampled systems