Are Convex Optimization Curves Convex?

Guy Barzilai, Ohad Shamir, Moslem Zamani

In this paper, we study when we might expect the optimization curve induced by gradient descent to be \emph{convex} -- precluding, for example, an initial plateau followed by a sha…