AXS: A framework for fast astronomical data processing based on Apache Spark

Abstract

We introduce AXS (Astronomy eXtensions for Spark), a scalable open-source astronomical data analysis framework built on Apache Spark, a widely used industry-standard engine for big data processing. Building on capabilities present in Spark, AXS aims to enable querying and analyzing almost arbitrarily large astronomical catalogs using familiar Python/AstroPy concepts, DataFrame APIs, and SQL statements. We achieve this by i) adding support to Spark for efficient on-line positional cross-matching and ii) supplying a Python library supporting commonly-used operations for astronomical data analysis. To support scalable cross-matching, we developed a variant of the ZONES algorithm (Gray et al. 2004) capable of operating in distributed, shared-nothing architecture. We couple this to a data partitioning scheme that enables fast catalog cross-matching and handles the data skew often present in deep all-sky data sets. The cross-match and other often-used functionalities are exposed to the end users through an easy-to-use Python API. We demonstrate AXS' technical and scientific performance on SDSS, ZTF, Gaia DR2, and AllWise catalogs. Using AXS we were able to perform on-the-fly cross-match of Gaia DR2 (1.8 billion rows) and AllWise (900 million rows) data sets in ~ 30 seconds. We discuss how cloud-ready distributed systems like AXS provide a natural way to enable comprehensive end-user analyses of large datasets such as LSST.

References in corpus (18)

• The Zwicky Transient Facility: System Overview, Performance, and First Results
• The Palomar Transient Factory: System Overview, Performance and First Results
• The Zwicky Transient Facility: Science Objectives
• A Three-Dimensional Map of Milky-Way Dust
• Photometric Calibration of the First 1.5 Years of the Pan-STARRS1 Survey
• The DECam Plane Survey: Optical photometry of two billion objects in the southern Galactic plane
• A Synoptic Map of Halo Substructures from the Pan-STARRS1 3π Survey
• A Map of Dust Reddening to 4.5 kpc from Pan-STARRS1
• Hypercalibration: A Pan-STARRS1-based recalibration of the Sloan Digital Sky Survey
• Tracing the Orphan Stream to 55 kpc with RR Lyrae Stars
• catsHTM - A tool for fast accessing and cross-matching large astronomical catalogs
• Gaia Data Release 1. Cross-match with external catalogues - Algorithm and results
• A Gaia-PS1-SDSS (GPS1) Proper Motion Catalog Covering 3/4 of the Sky
• Mapping the Extinction Curve in 3D: Structure on Kiloparsec Scales
• Update on the Nature of Virgo Overdensity
• 3D Dust Mapping Reveals that Orion Forms Part of a Large Ring of Dust
• Two Distant Halo Velocity Groups Discovered by the Palomar Transient Factory
• Probabilistic Cross-Identification in Crowded Fields as an Assignment Problem

Cited by in corpus (6)

• Fink, a new generation of broker for the LSST community
• A Survey on Spark Ecosystem for Big Data Processing
• AstroCatR: a Mechanism and Tool for Efficient Time Series Reconstruction of Large-Scale Astronomical Catalogues
• HLC2: a highly efficient cross-matching framework for large astronomical catalogues on heterogeneous computing environments
• The Search for Planet and Planetesimal Transits of White Dwarfs with the Zwicky Transient Facility
• The Astronomy Commons Platform: A Deployable Cloud-Based Analysis Platform for Astronomy