Post-correlation radio frequency interference classification methods

Abstract

We describe and compare several post-correlation radio frequency interference classification methods. As data sizes of observations grow with new and improved telescopes, the need for completely automated, robust methods for radio frequency interference mitigation is pressing. We investigated several classification methods and find that, for the data sets we used, the most accurate among them is the SumThreshold method. This is a new method formed from a combination of existing techniques, including a new way of thresholding. This iterative method estimates the astronomical signal by carrying out a surface fit in the time-frequency plane. With a theoretical accuracy of 95% recognition and an approximately 0.1% false probability rate in simple simulated cases, the method is in practice as good as the human eye in finding RFI. In addition it is fast, robust, does not need a data model before it can be executed and works in almost all configurations with its default parameters. The method has been compared using simulated data with several other mitigation techniques, including one based upon the singular value decomposition of the time-frequency matrix, and has shown better results than the rest.

14 pages, 12 figures (11 in colour). The software that was used in the article can be downloaded from http://www.astro.rug.nl/rfi-software/

References in corpus (9)

• Foreground simulations for the LOFAR - Epoch of Reionization Experiment
• Foregrounds for observations of the cosmological 21 cm line: I. First Westerbork measurements of Galactic emission at 150 MHz in a low latitude field
• Fast Large-Scale Reionization Simulations
• Multichannel interference mitigation methods in radio astronomy
• Removing radio interference from contaminated astronomical spectra using an independent reference signal and closure relations
• The GMRT EoR Experiment: Limits on Polarized Sky Brightness at 150 MHz
• Radio Frequency Interference Mitigation at the WSRT
• Statistically Stable Estimates of Variance in Radioastronomical Observations as Tools for RFI Mitigation
• A new approach to mitigation of radio frequency interference in interferometric data

Cited by in corpus (174)

• LOFAR: The LOw-Frequency ARray
• CASA, the Common Astronomy Software Applications for Radio Astronomy
• WSClean: an implementation of a fast, generic wide-field imager for radio astronomy
• LOFAR facet calibration
• LOFAR 150-MHz observations of the Boötes field: Catalogue and Source Counts
• Data Analysis for Precision 21 cm Cosmology
• Design and characterization of the Large-Aperture Experiment to Detect the Dark Age (LEDA) radiometer systems
• First Season MWA EoR Power Spectrum Results at Redshift 7
• The low-frequency environment of the Murchison Widefield Array: radio-frequency interference analysis and mitigation
• Improving the Epoch of Reionization Power Spectrum Results from Murchison Widefield Array Season 1 Observations
• Probing Ionospheric Structures using the LOFAR radio telescope
• The 1.28 GHz MeerKAT DEEP2 Image
• Bayesian constraints on the global 21-cm signal from the Cosmic Dawn
• M87 at metre wavelengths: the LOFAR picture
• Systematic biases in low frequency radio interferometric data due to calibration: the LOFAR EoR case
• The first power spectrum limit on the 21-cm signal of neutral hydrogen during the Cosmic Dawn at z=20-25 from LOFAR
• No precise localization for FRB 150418: claimed radio transient is AGN variability
• Trends in Ultracool Dwarf Magnetism. I. X-Ray Suppression and Radio Enhancement
• Studying Galactic interstellar turbulence through fluctuations in synchrotron emission: First LOFAR Galactic foreground detection
• Deep VLA observations of the cluster 1RXS J0603.3+4214 in the frequency range 1-2 GHz
• First Season MWA Phase II EoR Power Spectrum Results at Redshift 7
• Deep radio observations of the radio halo of the bullet cluster 1E 0657-55.8
• The Importance of Wide-field Foreground Removal for 21 cm Cosmology: A Demonstration With Early MWA Epoch of Reionization Observations
• The LOFAR radio environment
• The Radio Relics and Halo of El Gordo, a Massive $z=0.870$ Cluster Merger
• Linear polarization structures in LOFAR observations of the interstellar medium in the 3C196 field
• Deep Very Large Array observations of the merging cluster CIZA J2242.8+5301: continuum and spectral imaging
• Limits on Fast Radio Bursts and other transient sources at 182 MHz using the Murchison Widefield Array
• HI intensity mapping with MeerKAT: Calibration pipeline for multi-dish autocorrelation observations
• Parametrising Epoch of Reionization foregrounds: A deep survey of low-frequency point-source spectra with the MWA
• First LOFAR observations at very low frequencies of cluster-scale non-thermal emission: the case of Abell 2256
• The Murchison Widefield Array 21 cm Power Spectrum Analysis Methodology
• LOFAR MSSS: Detection of a low-frequency radio transient in 400 hrs of monitoring of the North Celestial Pole
• LOFAR imaging of Cygnus A -- Direct detection of a turnover in the hotspot radio spectra
• Discovery of HI gas in a young radio galaxy at $z=0.44$ using the Australian Square Kilometre Array Pathfinder
• Broadband Radio Polarimetry and Faraday Rotation of 563 Extragalactic Radio Sources
• Low Frequency Radio Constraints on the Synchrotron Cosmic Web
• GASP XVII. HI imaging of the jellyfish galaxy JO206: gas stripping and enhanced star formation
• Real-time detection of an extreme scattering event: constraints on Galactic plasma lenses
• Absolving the SSINS of Precision Interferometric Radio Data: A New Technique for Mitigating Faint Radio Frequency Interference
• Chandra and JVLA observations of HST Frontier Fields cluster MACS J0717.5+3745
• A high-resolution mosaic of the neutral hydrogen in the M81 triplet
• The widest-frequency radio relic spectra: observations from 150 MHz to 30 GHz
• The impact of interference excision on 21-cm Epoch of Reionization power spectrum analyses
• FR II radio galaxies at low frequencies I: morphology, magnetic field strength and energetics
• Dissecting Galaxies: Separating Star Formation, Shock Excitation and AGN Activity in the Central Region of NGC 613
• Anatomy of a Cooling Flow: The Feedback Response to Pure Cooling in the Core of the Phoenix Cluster
• FLAGCAL:A flagging and calibration package for radio interferometric data
• The Low-Frequency Radio Eclipses of the Black Widow Pulsar J1810+1744
• The Tianlai Dish Pathfinder Array: design, operation and performance of a prototype transit radio interferometer
• Modelling of the Spectral Energy Distribution of Fornax A: Leptonic and Hadronic Production of High Energy Emission from the Radio Lobes
• Frontier Fields Clusters: Chandra and JVLA View of the Pre-Merging Cluster MACS J0416.1-2403
• Deep residual detection of radio frequency interference for FAST
• Simultaneous Multiwavelength Observations of Magnetic Activity in Ultracool Dwarfs. IV. The Active, Young Binary NLTT 33370 AB (=2MASS J13142039+1320011)
• The Rotation Period and Magnetic Field of the T Dwarf 2MASSI J1047539+212423 Measured From Periodic Radio Bursts
• LOFAR reveals the giant: a low-frequency radio continuum study of the outflow in the nearby FR I radio galaxy 3C 31
• IQRM: real-time adaptive RFI masking for radio transient and pulsar searches
• Post-correlation filtering techniques for off-axis source and RFI removal
• Radio Observations of HD 80606 Near Planetary Periastron
• The first Murchison Widefield Array low frequency radio observations of cluster scale non-thermal emission: the case of Abell 3667
• Robust Radio Interferometric Calibration Using the t-Distribution
• The AARTFAAC Cosmic Explorer: observations of the 21-cm power spectrum in the EDGES absorption trough
• First SETI Observations with China's Five-hundred-meter Aperture Spherical radio Telescope (FAST)
• A Late-time Radio Survey of Short GRBs at $z<0.5$: New Constraints on the Remnants of Neutron Star Mergers
• Variable and polarized radio emission from the T6 brown dwarf WISEP J112254.73+255021.5
• Illuminating the past 8 billion years of cold gas towards two gravitationally lensed quasars
• Deep Learning improves identification of Radio Frequency Interference
• MC${}^2$: Multi-wavelength and dynamical analysis of the merging galaxy cluster ZwCl 0008.8+5215: An older and less massive Bullet Cluster
• SERPent: Automated reduction and RFI-mitigation software for e-MERLIN
• Deep low-frequency radio observations of Abell 2256 I: The filamentary radio relic
• Detailed study of ELAIS N1 field with the uGMRT -- II. Source Properties and Spectral Variation Of Foreground Power Spectrum from 300-500 MHz Observations
• Carbon and hydrogen radio recombination lines from the cold clouds towards Cassiopeia A
• Polarization leakage in epoch of reionization windows: II. Primary beam model and direction dependent calibration
• Low-radio-frequency eclipses of the redback pulsar J2215+5135 observed in the image plane with LOFAR
• Spectral energy distribution and radio halo of NGC 253 at low radio frequencies
• New methods to constrain the radio transient rate: results from a survey of four fields with LOFAR
• The First Millimeter Detection of a Non-Accreting Ultracool Dwarf
• Wide-field LOFAR-LBA power-spectra analyses: Impact of calibration, polarization leakage and ionosphere
• LOFAR low-band antenna observations of the 3C295 and Bootes fields: source counts and ultra-steep spectrum sources
• A High Reliability Survey of Discrete Epoch of Reionization Foreground Sources in the MWA EoR0 Field
• Radio Frequency Interference Mitigation based on the ArPLS and SumThreshold Method
• Downstream depolarization in the Sausage relic: a 1-4 GHz Very Large Array study
• The early afterglow of GRB 190829A
• Implementing RFI mitigation in Radio Science
• Broadband radio polarimetry of Fornax A, I: Depolarized patches generated by advected thermal material from NGC 1316
• A numerical study of 21-cm signal suppression and noise increase in direction-dependent calibration of LOFAR data
• Learning to detect RFI in radio astronomy without seeing it
• The XXL Survey XVIII. ATCA 2.1 GHz radio source catalogue and source counts for the XXL-South field
• The best of both worlds: Combining LOFAR and Apertif to derive resolved radio spectral index images
• New EoR Power Spectrum Limits From MWA Phase II Using the Delay Spectrum Method and Novel Systematic Rejection
• The giant lobes of Centaurus A observed at 118 MHz with the Murchison Widefield Array
• Spectral index of the Galactic foreground emission in the 50-87 MHz range
• The ATLAS 5.5 GHz Survey of the Extended Chandra Deep Field South: The Second Data Release
• 21 new long-term variables in the GX 339-4 field: two years of MeerKAT monitoring
• The discovery of lensed radio and X-ray sources behind the Frontier Fields cluster MACS J0717.5+3745 with the JVLA and Chandra
• LOFAR 150-MHz observations of SS 433 and W 50
• LOFAR observations of decameter carbon radio recombination lines towards Cassiopeia A
• The All-Sky SignAl Short-Spacing INterferometer (ASSASSIN) I: Global sky measurements with the Engineering Development Array-2
• A search for radio emission from double-neutron star merger GW190425 using Apertif
• LOFAR early-time search for coherent radio emission from Short GRB 181123B
• Neutral hydrogen gas, past and future star-formation in galaxies in and around the 'Sausage' merging galaxy cluster
• HIDE & SEEK: End-to-End Packages to Simulate and Process Radio Survey Data
• Data Processing Pipeline For Tianlai Experiment
• A LOFAR RFI detection pipeline and its first results
• LOFAR early-time search for coherent radio emission from GRB 180706A
• LOFAR detections of low-frequency radio recombination lines towards Cassiopeia A
• Limits on radio emission from meteors using the MWA
• Discovery of Carbon Radio Recombination Lines in absorption towards Cygnus~A
• Diffuse radio emission in the galaxy cluster SPT-CL J2031-4037: a steep spectrum intermediate radio halo?
• MIGHTEE-HI: deep spectral line observations of the COSMOS field
• Precision astrometry of pulsars and other compact radio sources in the globular cluster M15
• The Spatial Correlation of Bent-Tail Galaxies and Galaxy Clusters
• Detecting Radio Frequency Interference in radio-antenna arrays with the Recurrent Neural Network algorithm
• LOFAR: Recent imaging results & future prospects
• The brightness and spatial distributions of terrestrial radio sources
• Blazar jet evolution revealed by multi-epoch broadband radio polarimetry
• The search for radio emission from exoplanets using LOFAR beam-formed observations: Jupiter as an exoplanet
• Mitigating radio frequency interference in CHIME/FRB real-time intensity data
• A Pilot Radio Search for Magnetic Activity in Directly Imaged Exoplanets
• WISEP J060738.65+242953.4: A Nearby. Pole-On L8 Brown Dwarf with Radio Emission
• Detecting the HI Power Spectrum in the Post-Reionization Universe with SKA-Low
• A FPGA-based Fast Converging Digital Adaptive Filter for Real-time RFI Mitigation on Ground Based Radio Telescopes
• Towards 21-cm Intensity Mapping at $z=2.28$ with uGMRT using the Tapered Gridded Estimator I: Foreground Avoidance
• Modelling the environment around five ultracool dwarfs via the radio domain
• Exploring the consequences of chromatic data excision in 21-cm Epoch of Reionization power spectrum observations
• Bayesian approach to radio frequency interference mitigation
• LOFAR observations of 4C+19.44. On the discovery of low frequency spectral curvature in relativistic jet knots
• MeerKAT-16 HI observation of the dIrr galaxy WLM
• The XXL Survey XI: ATCA 2.1 GHz continuum observations
• Apercal -- The Apertif Calibration Pipeline
• An Eigenvector-based Method of Radio Array Calibration and Its Application to the Tianlai Cylinder Pathfinder
• Very Large Array observations of the mini-halo and AGN feedback in the Phoenix cluster
• Discovery of large-scale diffuse radio emission in low-mass galaxy cluster Abell 1931
• A First Look for Molecules between 103 and 133 MHz using the Murchison Widefield Array
• A Matched Filter Technique For Slow Radio Transient Detection And First Demonstration With The Murchison Widefield Array
• MC^2: A deeper look at ZwCl 2341.1+0000 with Bayesian galaxy clustering and weak lensing analyses
• Faraday rotation study of NGC 612 (PKS 0131-36): a hybrid radio source and its magnetised circumgalactic environment
• Radio-loud fraction of z>6 quasars
• Towards $21$-cm intensity mapping at $z=2.28$ with uGMRT using the tapered gridded estimator III: Foreground removal
• Low-frequency radio observations of recurrent nova RS Ophiuchi with MeerKAT and LOFAR
• LOFAR 144-MHz follow-up observations of GW170817
• Timing calibration and spectral cleaning of LOFAR time series data
• Anomaly Detection and Radio-frequency Interference Classification with Unsupervised Learning in Narrowband Radio Technosignature Searches
• Reaching thermal noise at ultra-low radio frequencies: the Toothbrush radio relic downstream of the shock front
• Categorize Radio Interference using component and temporal analysis
• Towards $21$-cm intensity mapping at $z=2.28$ with uGMRT using the tapered gridded estimator -- IV. Wideband analysis
• Multi-Level Pre-Correlation RFI Flagging for Real-Time Implementation on UniBoard
• First upper limits on the 21-cm signal power spectrum of neutral hydrogen at $z=9.16$ from the LOFAR 3C196 field
• A Dictionary Approach to Identifying Transient RFI
• Two procedures to flag radio frequency interference in the UV plane
• Unveiling the Universe with Emerging Cosmological Probes
• The interstellar medium in the environment of the supernova-less long-duration GRB 111005A
• The Carina Nebula and Gum 31 molecular complex III: The distribution of the 1-3 GHz radio continuum across the whole nebula
• SuperCLASS -- I. The Super CLuster Assisted Shear Survey: Project overview and Data Release 1
• RFI Flagging in Solar and Space Weather Low Frequency Radio Observations
• A Multi-Frequency Study of the Milky Way-like Spiral Galaxy NGC 6744
• The significance of low frequency interferometric observations for the GPS pulsar flux estimation: the case of J1740+1000
• Study of systematics effects on the Cross Power Spectrum of 21 cm Line and Cosmic Microwave Background using Murchison Widefield Array Data
• The Nature of the Stingray Nebula from Radio Observations
• HI Intensity Mapping with the MIGHTEE Survey: First Results of the HI Power Spectrum
• A LOFAR prompt search for radio emission accompanying X-ray flares in GRB 210112A
• Strong low-frequency radio flaring from Cygnus X-3 observed with LOFAR
• Radio multifrequency observations of galaxy clusters. The Abell 399$-$401 pair
• Performance of the Segment Anything Model in Various RFI/Events Detection in Radio Astronomy
• Epoch of Reionization Power Spectrum Limits from Murchison Widefield Array Data Targeted at EoR1 Field
• SuperCLASS -- II: Photometric Redshifts and Characteristics of Spatially-Resolved $\mu$Jy Radio Sources
• Noise Reduction Method for Radio Astronomy Single Station Observation Based on Wavelet Transform and Mathematical Morphology
• Weighted Stacking of Radio Images Affected by Noise and Interfering Signals
• Constraints on the properties of warm ionized gas from low-frequency hydrogen radio recombination lines
• Operational Non-identifiability of Single-epoch Low-rank RFI Mitigation: Controlled Failure-mode Analysis and HERA Evidence
• Foreground Subtraction with a Tensor-Based Oriented Singular Value Decomposition Method for HI Experiments
• The Impact of GSM towers in Radio Astronomy
• An Introduction to Single-Antenna Radio Astronomical Polarimetry
• An Overview of CHIME, the Canadian Hydrogen Intensity Mapping Experiment