Large-scale text processing pipeline with Apache Spark

Alexey Svyatkovskiy, Kosuke Imai, Mary Kroeger +1

In this paper, we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the Unite…