51 citations · 202 across the 34 of their papers we have counts for
40 papers
One-step differentiation of iterative algorithms
Jérôme Bolte, Edouard Pauwels, Samuel Vaiter
In appropriate frameworks, automatic differentiation is transparent to the user at the cost of being a significant computational burden when the number of operations is large. For…
On the nature of Bregman functions
Edouard Pauwels
Let C be convex, compact, with nonempty interior and h be Legendre with domain C, continuous on C. We prove that h is Bregman if and only if it is strictly convex on C and C is a p…
Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problems
Jérôme Bolte, Edouard Pauwels, Antonio Silveti-Falls
We leverage path differentiability and a recent result on nonsmooth implicit differentiation calculus to give sufficient conditions ensuring that the solution to a monotone inclusi…
The derivatives of Sinkhorn-Knopp converge
Edouard Pauwels, Samuel Vaiter
We show that the derivatives of the Sinkhorn-Knopp algorithm, or iterative proportional fitting procedure, converge towards the derivatives of the entropic regularization of the op…
On the complexity of nonsmooth automatic differentiation
Jérôme Bolte, Ryan Boustany, Edouard Pauwels +1
Using the notion of conservative gradient, we provide a simple model to estimate the computational costs of the backward and forward modes of algorithmic differentiation for a wide…
Automatic differentiation of nonsmooth iterative algorithms
Jérôme Bolte, Edouard Pauwels, Samuel Vaiter
Differentiation along algorithms, i.e., piggyback propagation of derivatives, is now routinely used to differentiate iterative solvers in differentiable programming. Asymptotics is…
Subgradient sampling for nonsmooth nonconvex minimization
Jérôme Bolte, Tam Le, Edouard Pauwels
Risk minimization for nonsmooth nonconvex problems naturally leads to first-order sampling or, by an abuse of terminology, to stochastic subgradient descent. We establish the conve…
Path differentiability of ODE flows
Swann Marx, Edouard Pauwels
We consider flows of ordinary differential equations (ODEs) driven by path differentiable vector fields. Path differentiable functions constitute a proper subclass of Lipschitz fun…
Semialgebraic Representation of Monotone Deep Equilibrium Models and Applications to Certification
Tong Chen, Jean-Bernard Lasserre, Victor Magron +1
Deep equilibrium models are based on implicitly defined functional relations and have shown competitive performance compared with the traditional deep networks. Monotone operator e…
Nonsmooth Implicit Differentiation for Machine Learning and Optimization
Jérôme Bolte, Tam Le, Edouard Pauwels +1
In view of training increasingly complex learning architectures, we establish a nonsmooth implicit function theorem with an operational calculus. Our result applies to most practic…
Conservative parametric optimality and the ridge method for tame min-max problems
Edouard Pauwels
We study the ridge method for min-max problems, and investigate its convergence without any convexity, differentiability or qualification assumption. The central issue is to determ…
Second-order step-size tuning of SGD for non-convex optimization
Camille Castera, Jérôme Bolte, Cédric Févotte +1
In view of a direct and simple improvement of vanilla SGD, this paper presents a fine-tuning of its step-sizes in the mini-batch case. For doing so, one estimates curvature, based…
A Sublevel Moment-SOS Hierarchy for Polynomial Optimization
Tong Chen, Jean-Bernard Lasserre, Victor Magron +1
We introduce a sublevel Moment-SOS hierarchy where each SDP relaxation can be viewed as an intermediate (or interpolation) between the d-th and (d+1)-th order SDP relaxations of th…
Sequential convergence of AdaGrad algorithm for smooth convex optimization
Cheik Traoré, Edouard Pauwels
We prove that the iterates produced by, either the scalar step size variant, or the coordinatewise variant of AdaGrad algorithm, are convergent sequences when applied to convex obj…
A Hölderian backtracking method for min-max and min-min problems
Jérôme Bolte, Lilian Glaudin, Edouard Pauwels +1
We present a new algorithm to solve min-max or min-min problems out of the convex world. We use rigidity assumptions, ubiquitous in learning, making our method applicable to many o…
Incremental Without Replacement Sampling in Nonconvex Optimization
Edouard Pauwels
Minibatch decomposition methods for empirical risk minimization are commonly analysed in a stochastic approximation setting, also known as sampling with replacement. On the other h…
A mathematical model for automatic differentiation in machine learning
Jerome Bolte, Edouard Pauwels
Automatic differentiation, as implemented today, does not have a simple mathematical model adapted to the needs of modern machine learning. In this work we articulate the relations…
Long term dynamics of the subgradient method for Lipschitz path differentiable functions
Jerome Bolte, Edouard Pauwels, Rodolfo Rios-Zertuche
We consider the long-term dynamics of the vanishing stepsize subgradient method in the case when the objective function is neither smooth nor convex. We assume that this function i…
Semialgebraic Optimization for Lipschitz Constants of ReLU Networks
Tong Chen, Jean-Bernard Lasserre, Victor Magron +1
The Lipschitz constant of a network plays an important role in many applications of deep learning, such as robustness certification and Wasserstein Generative Adversarial Network.…
Curiosities and counterexamples in smooth convex optimization
Jerome Bolte, Edouard Pauwels
Counterexamples to some old-standing optimization problems in the smooth convex coercive setting are provided. We show that block-coordinate, steepest descent with exact search or…
Rate of convergence for geometric inference based on the empirical Christoffel function
Mai Trang Vu, François Bachoc, Edouard Pauwels
We consider the problem of estimating the support of a measure from a finite, independent, sample. The estimators which are considered are constructed based on the empirical Christ…
Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning
Jérôme Bolte, Edouard Pauwels
Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we d…
An Inertial Newton Algorithm for Deep Learning
Camille Castera, Jérôme Bolte, Cédric Févotte +1
We introduce a new second-order inertial optimization method for machine learning called INNA. It exploits the geometry of the loss function while only requiring stochastic approxi…
Semi-algebraic approximation using Christoffel-Darboux kernel
Swann Marx, Edouard Pauwels, Tillmann Weisser +2
We provide a new method to approximate a (possibly discontinuous) function using Christoffel-Darboux kernels. Our knowledge about the unknown multivariate function is in terms of f…