DiFX: A software correlator for very long baseline interferometry using multi-processor computing environments

Abstract

We describe the development of an FX style correlator for Very Long Baseline Interferometry (VLBI), implemented in software and intended to run in multi-processor computing environments, such as large clusters of commodity machines (Beowulf clusters) or computers specifically designed for high performance computing, such as multi-processor shared-memory machines. We outline the scientific and practical benefits for VLBI correlation, these chiefly being due to the inherent flexibility of software and the fact that the highly parallel and scalable nature of the correlation task is well suited to a multi-processor computing environment. We suggest scientific applications where such an approach to VLBI correlation is most suited and will give the best returns. We report detailed results from the Distributed FX (DiFX) software correlator, running on the Swinburne supercomputer (a Beowulf cluster of approximately 300 commodity processors), including measures of the performance of the system. For example, to correlate all Stokes products for a 10 antenna array, with an aggregate bandwidth of 64 MHz per station and using typical time and frequency resolution presently requires of order 100 desktop-class compute nodes. Due to the effect of Moore's Law on commodity computing performance, the total number and cost of compute nodes required to meet a given correlation task continues to decrease rapidly with time. We show detailed comparisons between DiFX and two existing hardware-based correlators: the Australian Long Baseline Array (LBA) S2 correlator, and the NRAO Very Long Baseline Array (VLBA) correlator. In both cases, excellent agreement was found between the correlators. Finally, we describe plans for the future operation of DiFX on the Swinburne supercomputer, for both astrophysical and geodetic science.

41 pages, 10 figures, accepted for publication in PASP

References in corpus (1)

• A Very High Spectral Resolution Study of Ground-State OH Masers in W3(OH)

Cited by in corpus (180)

• First M87 Event Horizon Telescope Results. I. The Shadow of the Supermassive Black Hole
• The Murchison Widefield Array: the Square Kilometre Array Precursor at low radio frequencies
• DiFX2: A more flexible, efficient, robust and powerful software correlator
• Science with ASKAP - the Australian Square Kilometre Array Pathfinder
• ON the Nature of the Local Spiral Arm of the Milky Way
• The limb-brightened jet of M87 down to 7 Schwarzschild radii scale
• Fermi Large Area Telescope View of the Core of the Radio Galaxy Centaurus A
• Parallaxes for W49N and G048.60+0.02: Distant Star Forming Regions in the Perseus Spiral Arm
• 100 Microarcsecond Resolution VLBI Imaging of Anisotropic Interstellar Scattering towards Pulsar B0834+06
• Trigonometric parallaxes of star forming regions in the Sagittarius spiral arm
• Microarcsecond VLBI pulsar astrometry with PSR$\pi$ II. parallax distances for 57 pulsars
• Disc-jet coupling in the 2009 outburst of the black hole candidate H1743-322
• The SFXC software correlator for Very Long Baseline Interferometry: Algorithms and Implementation
• Event Horizon Telescope observations of the jet launching and collimation in Centaurus A
• The Commensal Real-time ASKAP Fast Transients (CRAFT) survey
• The Parallax of W43: a Massive Star Forming Complex near the Galactic Bar
• Extremely high precision VLBI astrometry of PSR J0437-4715 and implications for theories of gravity
• The European Pulsar Timing Array: current efforts and a LEAP toward the future
• A parallax distance and mass estimate for the transitional millisecond pulsar system J1023+0038
• Trigonometric Parallaxes of Star Forming Regions in the Perseus Spiral Arm
• TANAMI: Tracking Active Galactic Nuclei with Austral Milliarcsecond Interferometry I. First-Epoch 8.4 GHz Images
• Precision southern hemisphere VLBI pulsar astrometry II: Measurement of seven parallaxes
• LEAP: the large European array for pulsars
• The Angular Broadening of the Galactic Center Pulsar SGR J1745-29: A New Constraint on the Scattering Medium
• Trigonometric parallaxes to star-forming regions within 4 kpc of the galactic center
• A real-time software backend for the GMRT
• TANAMI monitoring of Centaurus A: The complex dynamics in the inner parsec of an extragalactic jet
• Trigonometric Parallaxes of Star Forming Regions in the Scutum Spiral Arm
• The Sardinia Radio Telescope: From a Technological Project to a Radio Observatory
• Mapping Spiral Structure on the far side of the Milky Way
• Parsec-Scale Radio Emission from the Low-Luminosity Active Galactic Nucleus in the Dwarf Starburst Galaxy Henize 2-10
• Trigonometric Parallaxes of Massive Star Forming Regions: G012.88+0.48 and W33
• rPICARD: A CASA-based Calibration Pipeline for VLBI Data
• The UTMOST: A hybrid digital signal processor transforms the MOST
• A heat-wave of accretion energy traced by masers in the G358-MM1 high-mass protostar
• Location of Gamma-ray emission and magnetic field strengths in OJ 287
• The ALMA Phasing System: A Beamforming Capability for Ultra-High-Resolution Science at (Sub)Millimeter Wavelengths
• Radio monitoring of the hard state jets in the 2011 outburst of MAXI J1836-194
• TANAMI Blazars in the IceCube PeV Neutrino Fields
• The Megamaser Cosmology Project.IX. Black hole masses for three maser galaxies
• mJIVE-20: a survey for compact mJy radio objects with the Very Long Baseline Array
• The wide-field VLBA calibrator survey -- WFCS
• V-FASTR: The VLBA Fast Radio Transients Experiment
• Exploring the Transient Radio Sky with VLITE: Early Results
• Wide-field VLBA Observations of the Chandra Deep Field South
• An accurate geometric distance to the compact binary SS Cygni vindicates accretion disc theory
• Kiloparsec-scale jets in three radio-loud narrow-line Seyfert 1 galaxies
• Discovery of extended and variable radio structure from the gamma-ray binary system PSR B1259-63/LS 2883
• Mosaiced wide-field VLBI observations of the Lockman Hole/XMM
• The faint radio sky: VLBA observations of the COSMOS field
• The geometric distance and binary orbit of PSR B1259-63
• The Radio Fundamental Catalogue. I. Astrometry
• Multi-wavelength view of the close-by GRB 190829A sheds light on gamma-ray burst physics
• Calibration of mixed-polarization interferometric observations
• The AuScope Geodetic VLBI Array
• The proper motion of PSR J1550-5418 measured with VLBI: a second magnetar velocity measurement
• The Sub-parsec Scale Radio Properties of Southern Starburst Galaxies. II. Supernova Remnants, the Supernova Rate, and the Ionised Medium in the NGC 4945 Starburst
• Dual-frequency VLBI study of Centaurus A on sub-parsec scales
• Resolved, expanding jets in the Galactic black hole candidate XTE J1908+094
• Delayed triggering of radio Active Galactic Nuclei in gas-rich minor mergers in the local Universe
• Cobalt: A GPU-based correlator and beamformer for LOFAR
• Fermi/LAT discovery of gamma-ray emission from the flat-spectrum radio quasar PKS 1454-354
• VLBI astrometry of PSR J2222-0137: a pulsar distance measured to 0.4% accuracy
• The First Very Long Baseline Interferometric SETI Experiment
• The Gamma-Ray Emitting Radio-Loud Narrow-Line Seyfert 1 Galaxy PKS 2004-447 II. The Radio View
• 50 pico arcsecond astrometry of pulsar emission
• Microarcsecond VLBI pulsar astrometry with PSRPI I. Two binary millisecond pulsars with white dwarf companions
• Swift J1727.8-1613 has the Largest Resolved Continuous Jet Ever Seen in an X-ray Binary
• A VLBA search for binary black holes in active galactic nuclei with double-peaked optical emission line spectra
• Nowhere to Hide: Radio-faint AGN in the GOODS-N field. I. Initial catalogue and radio properties
• Engineering and Science Highlights of the KAT-7 Radio Telescope
• Collimation of the relativistic jet in the quasar 3C 273
• A deep, high resolution survey of the low frequency radio sky
• Trigonometric Parallaxes of Star Forming Regions Beyond the Tangent Point of the Sagittarius Spiral Arm
• The unusual multiwavelength properties of the gamma-ray source PMNJ1603-4904
• RadioAstron space VLBI imaging of polarized radio emission in the high-redshift quasar 0642+449 at 1.6 GHz
• VLBI detection of an Infrared-Faint Radio Source
• Compact Bright Radio-loud AGNs -- III. A Large VLBA Survey at 43 GHz
• The Gamma-Ray Emitting Radio-Loud Narrow-Line Seyfert 1 Galaxy PKS 2004-447 I. The X-Ray View
• ALMA observations of PKS 1549-79: a case of feeding and feedback in a young radio quasar
• VLBI observations of the radio quasar J2228+0110 at z=5.95 and other field sources in multiple-phase-centre mode
• VLBI imaging throughout the primary beam using accurate UV shifting
• Unravelling the Innermost Jet Structure of OJ 287 with the First GMVA+ALMA Observations
• HI emission and absorption in nearby, gas-rich galaxies II. -- sample completion and detection of intervening absorption in NGC 5156
• Astrometric observations of X-ray binaries using very long baseline interferometry
• The First Very Long Baseline Interferometry Image of 44 GHz Methanol Maser with the KVN and VERA Array (KaVA)
• Detection of a Parsec-Scale Jet in a Radio-Quiet Narrow-Line Seyfert 1 Galaxy with Highly Accreting Supermassive Black Hole
• The environment of the strongest galactic methanol maser
• The EVN Galactic Plane Survey - EGaPS
• On the origin of LS 5039 and PSR J1825-1446
• CASA on the fringe -- Development of VLBI processing capabilities for CASA
• Astrometry of Cassini with the VLBA to Improve the Saturn Ephemeris
• Reversals in the Direction of Polarization Rotation in OJ 287
• Parallaxes for star forming regions in the inner Perseus spiral arm
• Parsec-scale Properties of eight Fanaroff-Riley Type 0 Radio Galaxies
• A novel search for gravitationally lensed radio sources in wide-field VLBI imaging from the mJIVE-20 survey
• The first resolved imaging of milliarcsecond-scale jets in Circinus X-1
• Astrometrically registered maps of H2O and SiO masers toward VX Sagittarii
• Precision Southern Hemisphere pulsar VLBI astrometry: techniques and results for PSR J1559-4438
• High Resolution Radio Astronomy Using Very Long Baseline Interferometry
• Detection of fast radio transients with multiple stations: a case study using the Very Long Baseline Array
• Imaging and variability studies of CTA~102 during the 2016 January $\gamma$-ray flare
• Radio observations of GRB 100418a: Test of an energy injection model explaining long-lasting GRB afterglows
• Limits on the event rates of fast radio transients from the V-FASTR experiment
• Astrometry of OH/IR stars using 1612 MHz hydroxyl masers. I. Annual parallaxes of WX Psc and OH138.0+7.2
• A VLBA Survey of radio stars in the Orion Nebula Cluster: I. The nonthermal radio population
• Multiband RadioAstron space VLBI imaging of the jet in quasar S5 0836+710
• The Parallax of the Red Hypergiant VX Sgr with Accurate Tropospheric Delay Calibration
• Disentangling interstellar plasma screens with pulsar VLBI: Combining auto- and cross-correlations
• High-resolution Observations of Low-luminosity Gigahertz-Peaked Spectrum and Compact Steep Spectrum Sources
• Reconciling optical and radio observations of the binary millisecond pulsar PSR J1640+2224
• Widefield VLBI observations of M31: A Unique Probe of the Ionized ISM of a nearby galaxy
• Effects of Intermittent Emission: Noise Inventory for Scintillating Pulsar B0834+06
• e-VLBI observations of GHz-Peaked Spectrum (GPS) radio sources in nearby galaxies from the AT20G survey
• Overview of the Observing System and Initial Scientific Accomplishments of the East Asian VLBI Network (EAVN)
• TANAMI: Tracking Active Galactic Nuclei with Austral Milliarcsecond Interferometry - II. Additional Sources
• Time-dependent visibility modelling of a relativistic jet in the X-ray binary MAXI J1803-298
• Short-baseline interferometry local-tie experiments at the Onsala Space Observatory
• Interferometric Monitoring of Gamma-ray Bright AGNs: OJ 287
• Parsec-scale shocks in the kiloparsec-scale jet of Centaurus A
• Interferometric Monitoring of Gamma-ray Bright AGNs: S5 0716+714
• Multi-Epoch Very Long Baseline Interferometric Observations of the Nuclear Starburst Region of NGC 253: Improved modelling of the supernova and star-formation rates
• High spectral resolution multi-tone Spacecraft Doppler tracking software: Algorithms and implementations
• Noise in the Cross-Power Spectrum of the Vela Pulsar
• The hyperluminous, dust-obscured quasar W2246-0526 at z=4.6: detection of parsec-scale radio activity
• Optical and radio astrometry of the galaxy associated with FRB150418
• Radio observations of SN2004dk with VLITE confirm late-time re-brightening
• Evolution of the pc-scale structure of PKS 1934-638 revisited: first science with the ASKAP and New Zealand telescopes
• VLBA+GBT observations of the COSMOS field and radio source counts at 1.4 GHz
• A broadband radio view of transient jet ejecta in the black hole candidate X-ray binary MAXI J1535-571
• e-VLBI observations of Circinus X-1: monitoring of the quiescent and flaring radio emission on AU scales
• A VLBA Trigonometric Parallax for RR Aql and the Mira PL Relation
• A VLBI survey of compact Broad Absorption Lines (BAL) quasars with BALnicity Index BI=0
• A broad HCO+ absorption line associated with the circumnuclear torus of NGC 1052
• The Most Compact Bright Radio-loud AGN -- II. VLBA Observations of Ten Sources at 43 and 86~GHz
• Absolute kinematics of radio-source components in the complete S5 polar cap sample. IV. Proper motions of the radio cores over a decade and spectral properties
• RadioAstron: An Earth-Space Radio Interferometer with a 350,000 km Baseline
• A Milliarcsecond-accurate Position for Sagittarius A*
• Active galactic nuclei cores in infrared-faint radio sources: Very long baseline interferometry observations using the Very Long Baseline Array
• VLBI Astrometry of Radio Stars to Link Radio and Optical Celestial Reference Frames. I. HD 199178 $\&$ AR Lacertae
• On the Structure of the Sagittarius Spiral Arm in the Inner Milky Way
• Near real-time astrometry for spacecraft navigation with the VLBA: A demonstration with the Mars Reconnaissance Orbiter and Odyssey
• An Interferometric SETI Observation of Kepler-111 b
• First observations with a GNSS antenna to radio telescope interferometer
• New timing measurement results of 16 pulsars
• Probing the ionosphere by the pulsar B0950+08 with help of RadioAstron ground-space baselines
• The NTSC VLBI System and its application in UT1 measurement
• Scaling Radio Astronomy Signal Correlation on Heterogeneous Supercomputers Using Various Data Distribution Methodologies
• First geodetic observations using new VLBI stations ASKAP-29 and WARK12M
• Circular polarization measurement in millimeter-wavelength spectral-line VLBI observations
• A Fast Radio Burst Search Method for VLBI Observation
• Real Time Event Detection in Astronomical Data Streams: Lessons from the VLBA
• Is Fermi 1544-0649 a misaligned blazar? discovering the jet structure with VLBI
• Very-long-baseline interferometry study of the flaring blazar TXS 1508+572 in the early Universe
• VLBA astrometry of PSRs B0329+54 and B1133+16: Improved pulsar distances and comparison of global ionospheric models
• High Resolution Rapid Response observations of compact radio sources with the Ceduna Hobart Interferometer (CHI)
• Optimal Correlation Estimators for Quantized Signals
• Radio follow-up of a candidate gamma-ray transient in the sky localization area of GW170608
• Revealing two radio active galactic nuclei extremely near PSR J0437$-$4715
• Prospects for Wideband VLBI Correlation in the Cloud
• Is 4C+29.48 a gamma-ray source?
• Archival VLBA observations of the Cygnus A Nuclear Radio Transient (Cyg A-2) Strengthen the Tidal Disruption Event Interpretation
• Broad band flux-density monitoring of radio sources with the Onsala twin telescopes
• VLBI astrometry of two millisecond pulsars
• rPICARD: A CASA-based Calibration Pipeline for VLBI Data. Calibration and imaging of 7 mm VLBA observations of the AGN jet in M87
• The localization of single pulse in VLBI observation
• The Multi-Epoch Jet Outbursts in Abell 496: synchrotron ageing and buoyant X-ray cavities draped by warm gas filaments
• A wider audience: Turning VLBI into a survey instrument
• The radio afterglow of the ultra-long GRB 220627A
• VLBI Observations of SN 2012au Reveal a Compact Radio Source a Decade Post Explosion
• Demonstration of KVN phase referencing capability
• Software correlators as testbeds for RFI algorithms
• A High-frequency Geodetic VLBI Experiment for Optical Clock Comparison
• Probing the innermost regions of AGN jets and their magnetic fields with RadioAstron. V. Space and ground millimeter-VLBI imaging of OJ 287
• Light Deflection under the Gravitational Field of Jupiter -- Testing General Relativity
• On the Nature of the Protoplanetary Nebula OH\,231.8+4.2
• Arbitrary Transform Telescopes: The Generalization of Interferometry
• Multiwavelength and parsec-scale properties of extragalactic jets
• A unified model of feed rotation in radio telescopes and GNSS antennas
• Heavy interstellar scattering toward the near end of the Galactic bar