Corral Framework: Trustworthy and Fully Functional Data Intensive Parallel Astronomical Pipelines

Abstract

Data processing pipelines represent an important slice of the astronomical software library that include chains of processes that transform raw data into valuable information via data reduction and analysis. In this work we present Corral, a Python framework for astronomical pipeline generation. Corral features a Model-View-Controller design pattern on top of an SQL Relational Database capable of handling: custom data models; processing stages; and communication alerts, and also provides automatic quality and structural metrics based on unit testing. The Model-View-Controller provides concept separation between the user logic and the data models, delivering at the same time multi-processing and distributed computing capabilities. Corral represents an improvement over commonly found data processing pipelines in Astronomy since the design pattern eases the programmer from dealing with processing flow and parallelization issues, allowing them to focus on the specific algorithms needed for the successive data transformations and at the same time provides a broad measure of quality over the created pipeline. Corral and working examples of pipelines that use it are available to the community at https://github.com/toros-astro.

arXiv comment19 pages, 3 figures, accepted for publication at Astronomy and Computing. Code available at https://github.com/toros-astro/corral

References in corpus (14)

• Scikit-learn: Machine Learning in Python
• Astropy: A Community Python Package for Astronomy
• The NumPy array: a structure for efficient numerical computation
• Binary Black Hole Mergers in the first Advanced LIGO Observing Run
• VISTA Variables in the Via Lactea (VVV): The public ESO near-IR variability survey of the Milky Way
• The Sloan Digital Sky Survey Monitor Telescope Pipeline
• The Exoplanet Orbit Database II: Updates to exoplanets.org
• The IPAC Image Subtraction and Discovery Pipeline for the intermediate Palomar Transient Factory
• Worldwide site comparison for submillimetre astronomy
• The Astronomer's Telegram: A Web-based Short-Notice Publication System for the Professional Astronomical Community
• RTS2 - the Remote Telescope System
• The Transient Optical Sky Survey Data Pipeline
• GONG ClassicMerge: Pipeline and Product
• A Prototype for the Cherenkov Telescope Array Pipelines Framework: Modular Efficiency Simple System (MESS)

Cited by in corpus (4)

• The Astropy Project: Sustaining and Growing a Community-oriented Open-source Project and the Latest Major Release (v5.0) of the Core Package
• Machine Learning on Difference Image Analysis: A comparison of methods for transient detection
• From Data Processes to Data Products: Knowledge Infrastructures in Astronomy
• Automatic Catalog of RRLyrae from $\sim$ 14 million VVV Light Curves: How far can we go with traditional machine-learning?