An Integer Programming Approach to Deep Neural Networks with Binary Activation Functions
arXiv:2007.03326
Abstract
We study deep neural networks with binary activation functions (BDNN), i.e. the activation function only has two states. We show that the BDNN can be reformulated as a mixed-integer linear program which can be solved to global optimality by classical integer programming solvers. Additionally, a heuristic solution algorithm is presented and we study the model under data uncertainty, applying a two-stage robust optimization approach. We implemented our methods on random and real datasets and show that the heuristic version of the BDNN outperforms classical deep neural networks on the Breast Cancer Wisconsin dataset while performing worse on random data.
References in corpus (8)
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Binary Neural Networks: A Survey
- Robustness and Regularization of Support Vector Machines
- Bitwise Neural Networks
- Sparse Regression: Scalable algorithms and empirical performance
- Sparse Classification: a scalable discrete optimization perspective
- Combinatorial Attacks on Binarized Neural Networks
Cited by in corpus (4)
- Optimal training of integer-valued neural networks with mixed integer programming
- Taming Binarized Neural Networks and Mixed-Integer Programs
- On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
- Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks