Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing
arXiv:2207.02026 · doi:10.14778/3551793.3551855
Abstract
Big data processing at the production scale presents a highly complex environment for resource optimization (RO), a problem crucial for meeting performance goals and budgetary constraints of analytical users. The RO problem is challenging because it involves a set of decisions (the partition count, placement of parallel instances on machines, and resource allocation to each instance), requires multi-objective optimization (MOO), and is compounded by the scale and complexity of big data systems while having to meet stringent time constraints for scheduling. This paper presents a MaxCompute-based integrated system to support multi-objective resource optimization via fine-grained instance-level modeling and optimization. We propose a new architecture that breaks RO into a series of simpler problems, new fine-grained predictive models, and novel optimization methods that exploit these models to make effective instance-level recommendations in a hierarchical MOO framework. Evaluation using production workloads shows that our new RO system could reduce 37-72% latency and 43-78% cost at the same time, compared to the current optimizer and scheduler, while running in 0.02-0.23s.
References in corpus (14)
- Graph Transformer Networks
- Neo: A Learned Query Optimizer
- BestConfig: Tapping the Performance Potential of Systems via Automatic Configuration Tuning
- Deep Unsupervised Cardinality Estimation
- Plan-Structured Deep Neural Network Models for Query Performance Prediction
- A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation
- Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning
- WiSeDB: A Learning-based Workload Management Advisor for Cloud Databases
- Tempo: Robust and Self-Tuning Resource Management in Multi-tenant Parallel Databases
- FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation
- ClassyTune: A Performance Auto-Tuner for Systems in the Cloud
- Flow-Loss: Learning Cardinality Estimates That Matter
- BayesCard: Revitilizing Bayesian Frameworks for Cardinality Estimation
- Phoebe: A Learning-based Checkpoint Optimizer