MLlib: Machine Learning in Apache Spark

Abstract

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.

References in corpus (1)

• TuPAQ: An Efficient Planner for Large-scale Predictive Analytic Queries

Cited by in corpus (54)

• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• Speeding Up Distributed Machine Learning Using Codes
• SCARFF: a Scalable Framework for Streaming Credit Card Fraud Detection with Spark
• What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)
• A Survey on Spark Ecosystem for Big Data Processing
• A Survey on Geographically Distributed Big-Data Processing using MapReduce
• SparkNet: Training Deep Networks in Spark
• LaraDB: A Minimalist Kernel for Linear and Relational Algebra Computation
• Query Processing on Tensor Computation Runtimes
• Machine Learning at Microsoft with ML .NET
• A Distributed and Incremental SVD Algorithm for Agglomerative Data Analysis on Large Networks
• End-to-end Optimization of Machine Learning Prediction Queries
• CFM-BD: a distributed rule induction algorithm for building Compact Fuzzy Models in Big Data classification problems
• Teaching Vehicles to Anticipate: A Systematic Study on Probabilistic Behavior Prediction Using Large Data Sets
• Event-based Failure Prediction in Distributed Business Processes
• Classification of large DNA methylation datasets for identifying cancer drivers
• Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues
• On the cross-validation bias due to unsupervised pre-processing
• An Experimental Evaluation of Large Scale GBDT Systems
• A Utilization Model for Optimization of Checkpoint Intervals in Distributed Stream Processing Systems
• From Server-Based to Client-Based Machine Learning: A Comprehensive Survey
• TopicBERT: A Transformer transfer learning based memory-graph approach for multimodal streaming social media topic detection
• A Tour of TensorFlow
• Modern Data Formats for Big Bioinformatics Data Analytics
• FairTest: Discovering Unwarranted Associations in Data-Driven Applications
• TensorFlow Estimators: Managing Simplicity vs. Flexibility in High-Level Machine Learning Frameworks
• SLAQ: Quality-Driven Scheduling for Distributed Machine Learning
• Splash: User-friendly Programming Interface for Parallelizing Stochastic Algorithms
• L1-Regularized Distributed Optimization: A Communication-Efficient Primal-Dual Framework
• A State Transition Model for Mobile Notifications via Survival Analysis
• Polynomial methods in statistical inference: theory and practice
• Sequence-to-sequence models for workload interference
• CLAMShell: Speeding up Crowds for Low-latency Data Labeling
• A unified framework to improve the interoperability between HPC and Big Data languages and programming models
• Curse of "Low" Dimensionality in Recommender Systems
• Data Analytics and Machine Learning Methods, Techniques and Tool for Model-Driven Engineering of Smart IoT Services
• Distributed Evolution Strategies with Multi-Level Learning for Large-Scale Black-Box Optimization
• Architectural Impact on Performance of In-memory Data Analytics: Apache Spark Case Study
• Workflow environments for advanced cyberinfrastructure platforms
• Decision-forest voting scheme for classification of rare classes in network intrusion detection
• Faster and Cheaper: Parallelizing Large-Scale Matrix Factorization on GPUs
• The Renoir Dataflow Platform: Efficient Data Processing without Complexity
• MPIgnite: An MPI-Like Language and Prototype Implementation for Apache Spark
• HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units
• On the usage of the probability integral transform to reduce the complexity of multi-way fuzzy decision trees in Big Data classification problems
• Matrix Computations and Optimization in Apache Spark
• Whiz: A Fast and Flexible Data Analytics System
• Improving the Learnability of Machine Learning APIs by Semi-Automated API Wrapping
• ZenLDA: An Efficient and Scalable Topic Model Training System on Distributed Data-Parallel Platform
• Analyzing billion-objects catalog interactively: Apache Spark for physicists
• Distributed Multi-Task Relationship Learning
• clusterNOR: A NUMA-Optimized Clustering Framework
• knor: A NUMA-Optimized In-Memory, Distributed and Semi-External-Memory k-means Library
• MPI-FAUN: An MPI-Based Framework for Alternating-Updating Nonnegative Matrix Factorization