Training Quantized Neural Networks to Global Optimality via Semidefinite Programming

Abstract

Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware. Although post-training quantization is well-studied, training optimal quantized NNs involves combinatorial non-convex optimization problems which appear intractable. In this work, we introduce a convex optimization strategy to train quantized NNs with polynomial activations. Our method leverages hidden convexity in two-layer neural networks from the recent literature, semidefinite lifting, and Grothendieck's identity. Surprisingly, we show that certain quantized NN problems can be solved to global optimality in polynomial-time in all relevant parameters via semidefinite relaxations. We present numerical examples to illustrate the effectiveness of our method.

arXiv commentv2: Minor edits in the text. The results are unchanged

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Compressing Deep Convolutional Networks using Vector Quantization
• Fixed Point Quantization of Deep Convolutional Networks
• Sketchy Decisions: Convex Low-Rank Matrix Optimization with Optimal Storage
• Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks
• Revealing the Structure of Deep Neural Networks via Convex Duality
• Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization
• Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time
• Fast Convex Quadratic Optimization Solvers with Adaptive Sketching-based Preconditioners