Pattern Matching for Self- Tuning of MapReduce Jobs

Abstract

In this paper, we study CPU utilization time patterns of several MapReduce applications. After extracting running patterns of several applications, they are saved in a reference database to be later used to tweak system parameters to efficiently execute unknown applications in future. To achieve this goal, CPU utilization patterns of new applications are compared with the already known ones in the reference database to find/predict their most probable execution patterns. Because of different patterns lengths, the Dynamic Time Warping (DTW) is utilized for such comparison; a correlation analysis is then applied to DTWs outcomes to produce feasible similarity patterns. Three real applications (WordCount, Exim Mainlog parsing and Terasort) are used to evaluate our hypothesis in tweaking system parameters in executing similar applications. Results were very promising and showed effectiveness of our approach on pseudo-distributed MapReduce platforms.

7 pages, previously published as "On Using Pattern Matching Algorithms in MapReduce Applications" at ISPA 2011. arXiv admin note: substantial text overlap with arXiv:1112.5505

Cited by in corpus (6)

• On Modeling Dependency between MapReduce Configuration Parameters and Total Execution Time
• Network Load Analysis and Provisioning of MapReduce Applications
• Statistical Regression to Predict Total Cumulative CPU Usage of MapReduce Jobs
• Statistical Analysis to Extract Effective Parameters on Overall Energy Consumption of Wireless Sensor Network (WSN)
• Distributed computing of Seismic Imaging Algorithms
• Thesis Report: Resource Utilization Provisioning in MapReduce