Efficient and Robust Machine Learning for Real-World Systems

Abstract

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation and the vision of the Internet-of-Things fuel the interest in resource efficient approaches. These approaches require a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. On top of this, it is crucial to treat uncertainty in a consistent manner in all but the simplest applications of machine learning systems. In particular, a desideratum for any real-world system is to be robust in the presence of outliers and corrupted data, as well as being `aware' of its limits, i.e.\ the system should maintain and provide an uncertainty estimate over its own predictions. These complex demands are among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology into every day's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. First we provide a comprehensive review of resource-efficiency in deep neural networks with focus on techniques for model size reduction, compression and reduced precision. These techniques can be applied during training or as post-processing and are widely used to reduce both computational complexity and memory footprint. As most (practical) neural networks are limited in their ways to treat uncertainty, we contrast them with probabilistic graphical models, which readily serve these desiderata by means of probabilistic inference. In that way, we provide an extensive overview of the current state-of-the-art of robust and efficient machine learning for real-world systems.

References in corpus (34)

• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Practical Bayesian Optimization of Machine Learning Algorithms
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Expectation Propagation for approximate Bayesian inference
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Weight Uncertainty in Neural Networks
• Deep Learning with Limited Numerical Precision
• Binarized Neural Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• Predicting Parameters in Deep Learning
• Tensorizing Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• Towards Accurate Binary Convolutional Neural Network
• Convolutional Neural Networks using Logarithmic Data Representation
• Bayesian Compression for Deep Learning
• Training and Inference with Integers in Deep Neural Networks
• Training deep neural networks with low precision multiplications
• Bayesian Dark Knowledge
• Neural Networks with Few Multiplications
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• An exploration of parameter redundancy in deep networks with circulant projections
• Maximum Margin Bayesian Networks