The Zwicky Transient Facility Alert Distribution System

Abstract

The Zwicky Transient Facility (ZTF) survey generates real-time alerts for optical transients, variables, and moving objects discovered in its wide-field survey. We describe the ZTF alert stream distribution and processing (filtering) system. The system uses existing open-source technologies developed in industry: Kafka, a real-time streaming platform, and Avro, a binary serialization format. The technologies used in this system provide a number of advantages for the ZTF use case, including (1) built-in replication, scalability, and stream rewind for the distribution mechanism; (2) structured messages with strictly enforced schemas and dynamic typing for fast parsing; and (3) a Python-based stream processing interface that is similar to batch for a familiar and user-friendly plug-in filter system, all in a modular, primarily containerized system. The production deployment has successfully supported streaming up to 1.2 million alerts or roughly 70 GB of data per night, with each alert available to a consumer within about 10 s of alert candidate production. Data transfer rates of about 80,000 alerts/minute have been observed. In this paper, we discuss this alert distribution and processing system, the design motivations for the technology choices for the framework, performance in production, and how this system may be generally suitable for other alert stream use cases, including the upcoming Large Synoptic Survey Telescope.

Published in PASP Focus Issue on the Zwicky Transient Facility (doi: 10.1088/1538-3873/aae904). 9 Pages, 2 Figures

References in corpus (3)

• A Wolf-Rayet-like progenitor of supernova SN 2013cu from spectral observations of a wind
• Strong Ultraviolet Pulse From a Newborn Type Ia Supernova
• iPTF14yb: The First Discovery of a GRB Afterglow Independent of a High-Energy Trigger

Cited by in corpus (18)

• The Zwicky Transient Facility: System Overview, Performance, and First Results
• The Zwicky Transient Facility: Data Processing, Products, and Archive
• The Zwicky Transient Facility: Science Objectives
• Candidate Electromagnetic Counterpart to the Binary Black Hole Merger Gravitational Wave Event S190521g
• The Zwicky Transient Facility: Surveys and Scheduler
• The GROWTH Marshal: A Dynamic Science Portal for Time-Domain Astronomy
• Machine Learning for the Zwicky Transient Facility
• The Koala: A Fast Blue Optical Transient with Luminous Radio Emission from a Starburst Dwarf Galaxy at $z=0.27$
• ZTF Early Observations of Type Ia Supernovae I: Properties of the 2018 Sample
• Optical follow-up of the neutron star-black hole mergers S200105ae and S200115j
• ZTF 18aaqeasu (SN 2018byg): A Massive Helium-shell Double Detonation on a Sub-Chandrasekhar Mass White Dwarf
• SN2019dge: a Helium-rich Ultra-Stripped Envelope Supernova
• 2900 square degree search for the optical counterpart of short gamma-ray burst GRB 180523B with the Zwicky Transient Facility
• Discovery of Super-Slow Rotating Asteroids with ATLAS and ZTF photometry
• Supernova Siblings and their Parent Galaxies in the Zwicky Transient Facility Bright Transient Surve
• Deep Drilling in the Time Domain with DECam: Survey Characterization
• AT 2020iko: a WZ Sge-type DN candidate with an anomalous precursor event
• Faintest of them all : ZTF 21aaoryiz/SN 2021fcg -- Discovery of an extremely low luminosity type Iax supernova