A Polynomial Time Algorithm for Learning Halfspaces with Tsybakov Noise

Abstract

We study the problem of PAC learning homogeneous halfspaces in the presence of Tsybakov noise. In the Tsybakov noise model, the label of every sample is independently flipped with an adversarially controlled probability that can be arbitrarily close to $1/2$ for a fraction of the samples. {\em We give the first polynomial-time algorithm for this fundamental learning problem.} Our algorithm learns the true halfspace within any desired accuracy $\epsilon$ and succeeds under a broad family of well-behaved distributions including log-concave distributions. Prior to our work, the only previous algorithm for this problem required quasi-polynomial runtime in $1/\epsilon$. Our algorithm employs a recently developed reduction \cite{DKTZ20b} from learning to certifying the non-optimality of a candidate halfspace. This prior work developed a quasi-polynomial time certificate algorithm based on polynomial regression. {\em The main technical contribution of the current paper is the first polynomial-time certificate algorithm.} Starting from a non-trivial warm-start, our algorithm performs a novel "win-win" iterative process which, at each step, either finds a valid certificate or improves the angle between the current halfspace and the true one. Our warm-start algorithm for isotropic log-concave distributions involves a number of analytic tools that may be of broader interest. These include a new efficient method for reweighting the distribution in order to recenter it and a novel characterization of the spectrum of the degree-$2$ Chow parameters.

References in corpus (10)

• Risk bounds for statistical learning
• Sever: A Robust Meta-Algorithm for Stochastic Optimization
• Recent Advances in Algorithmic High-Dimensional Robust Statistics
• Revisiting Perceptron: Efficient and Label-Optimal Learning of Halfspaces
• Efficient active learning of sparse halfspaces with arbitrary bounded noise
• Statistical-Query Lower Bounds via Functional Gradients
• Near-Optimal SQ Lower Bounds for Agnostically Learning Halfspaces and ReLUs under Gaussian Marginals
• Classification Under Misspecification: Halfspaces, Generalized Linear Models, and Connections to Evolvability
• Learning Halfspaces with Tsybakov Noise
• Non-Convex SGD Learns Halfspaces with Adversarial Label Noise

Cited by in corpus (4)

• Boosting in the Presence of Massart Noise
• Sample-Optimal PAC Learning of Halfspaces with Malicious Noise
• Improved Algorithms for Efficient Active Learning Halfspaces with Massart and Tsybakov noise
• Robust Learning under Strong Noise via SQs