Programming with Neural Surrogates of Programs

Abstract

Surrogates, models that mimic the behavior of programs, form the basis of a variety of development workflows. We study three surrogate-based design patterns, evaluating each in case studies on a large-scale CPU simulator. With surrogate compilation, programmers develop a surrogate that mimics the behavior of a program to deploy to end-users in place of the original program. Surrogate compilation accelerates the CPU simulator under study by $1.6\times$. With surrogate adaptation, programmers develop a surrogate of a program then retrain that surrogate on a different task. Surrogate adaptation decreases the simulator's error by up to $50\%$. With surrogate optimization, programmers develop a surrogate of a program, optimize input parameters of the surrogate, then plug the optimized input parameters back into the original program. Surrogate optimization finds simulation parameters that decrease the simulator's error by $5\%$ compared to the error induced by expert-set parameters. In this paper we formalize this taxonomy of surrogate-based design patterns. We further describe the programming methodology common to all three design patterns. Our work builds a foundation for the emerging class of workflows based on programming with surrogates of programs.

References in corpus (19)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• TensorFlow: A system for large-scale machine learning
• Overcoming catastrophic forgetting in neural networks
• Variational Inference: A Review for Statisticians
• Machine learning and the physical sciences
• Reconciling modern machine learning practice and the bias-variance trade-off
• Church: a language for generative models
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Verifiable Reinforcement Learning via Policy Extraction
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Active learning of deep surrogates for PDEs: Application to metasurface design
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Applying machine learning to improve simulations of a chaotic dynamical system using empirical error correction
• uops.info: Characterizing Latency, Throughput, and Port Usage of Instructions on Intel Microarchitectures
• Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks
• Are Transformers universal approximators of sequence-to-sequence functions?
• Black-Box Optimization with Local Generative Surrogates
• Towards Learning Convolutions from Scratch

Cited by in corpus (1)

• Turaco: Complexity-Guided Data Sampling for Training Neural Surrogates of Programs