A Survey on Spark Ecosystem for Big Data Processing

Abstract

With the explosive increase of big data in industry and academic fields, it is necessary to apply large-scale data processing systems to analysis Big Data. Arguably, Spark is state of the art in large-scale data computing systems nowadays, due to its good properties including generality, fault tolerance, high performance of in-memory data processing, and scalability. Spark adopts a flexible Resident Distributed Dataset (RDD) programming model with a set of provided transformation and action operators whose operating functions can be customized by users according to their applications. It is originally positioned as a fast and general data processing system. A large body of research efforts have been made to make it more efficient (faster) and general by considering various circumstances since its introduction. In this survey, we aim to have a thorough review of various kinds of optimization techniques on the generality and performance improvement of Spark. We introduce Spark programming model and computing system, discuss the pros and cons of Spark, and have an investigation and classification of various solving techniques in the literature. Moreover, we also introduce various data management and processing systems, machine learning algorithms and applications supported by Spark. Finally, we make a discussion on the open issues and challenges for large-scale in-memory data processing with Spark.

arXiv comment21 pages, 11 figures, technique report. in IEEE Transactions on Knowledge and Data Engineering (2020). arXiv admin note: text overlap with arXiv:1302.2966 by other authors

References in corpus (9)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• MLlib: Machine Learning in Apache Spark
• SCARFF: a Scalable Framework for Streaming Credit Card Fraud Detection with Spark
• Flare Prediction Using Photospheric and Coronal Image Data
• The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox
• Splash: User-friendly Programming Interface for Parallelizing Stochastic Algorithms
• AXS: A framework for fast astronomical data processing based on Apache Spark
• Approximate Distributed Joins in Apache Spark
• Approximate Stream Analytics in Apache Flink and Apache Spark Streaming

Cited by in corpus (2)

• Explainable machine learning multi-label classification of Spanish legal judgements
• Big Data application in congestion detection and classification using Apache spark