Principal Component Analysis with Noisy and/or Missing Data

Abstract

We present a method for performing Principal Component Analysis (PCA) on noisy datasets with missing values. Estimates of the measurement error are used to weight the input data such that compared to classic PCA, the resulting eigenvectors are more sensitive to the true underlying signal variations rather than being pulled by heteroskedastic measurement noise. Missing data is simply the limiting case of weight=0. The underlying algorithm is a noise weighted Expectation Maximization (EM) PCA, which has additional benefits of implementation speed and flexibility for smoothing eigenvectors to reduce the noise contribution. We present applications of this method on simulated data and QSO spectra from the Sloan Digital Sky Survey.

Accepted for publication in PASP; v2 with minor updates, mostly to bibliography

References in corpus (4)

• K-corrections and filter transformations in the ultraviolet, optical, and near infrared
• The metallicity of the most distant quasars
• Principal Component Analysis of the Time- and Position-Dependent Point Spread Function of the Advanced Camera for Surveys
• A data-driven model for spectra: Finding double redshifts in the Sloan Digital Sky Survey

Cited by in corpus (41)

• The Eighteenth Data Release of the Sloan Digital Sky Surveys: Targeting and First Spectra from SDSS-V
• The Spectroscopic Data Processing Pipeline for the Dark Energy Spectroscopic Instrument
• The stellar population structure of the Galactic disk
• Dark Quest. I. Fast and Accurate Emulation of Halo Clustering Statistics and Its Application to Galaxy Clustering
• The weirdest SDSS galaxies: results from an outlier detection algorithm
• Weighted principal component analysis: a weighted covariance eigendecomposition approach
• YARARA: Significant improvement of RV precision through post-processing of spectral time-series
• Two temperate super-Earths transiting a nearby late-type M dwarf
• Dynamic mode decomposition using a Kalman filter for parameter estimation
• Performance of the Quasar Spectral Templates for the Dark Energy Spectroscopic Instrument
• DESI Observations of the Andromeda Galaxy: Revealing the Immigration History of our Nearest Neighbor
• Standardizing Type Ia Supernova Absolute Magnitudes Using Gaussian Process Data Regression
• Keck Integral-Field Spectroscopy of M87 Reveals an Intrinsically Triaxial Galaxy and a Revised Black Hole Mass
• The dimensionality of stellar chemical space using spectra from the Apache Point Observatory Galactic Evolution Experiment
• The Physical Properties of Low Redshift FeLoBAL Quasars. I. Spectral Synthesis Analysis of the BAL Outflows using $SimBAL$
• Exploring the spectroscopic diversity of type Ia supernovae with DRACULA: a machine learning approach
• Value Added Catalog of physical properties of more than 1.3 million galaxies from the DESI Survey
• Archetype-Based Redshift Estimation for the Dark Energy Spectroscopic Instrument Survey
• Prior-Informed AGN-Host Spectral Decomposition Using PyQSOFit
• Point Spread Function Modelling for Wide Field Small Aperture Telescopes with a Denoising Autoencoder
• A metric space for type Ia supernova spectra
• Image simulations for gravitational lensing with SkyLens
• The Physical Properties of Low-Redshift FeLoBAL Quasars: II. The Rest-Frame Optical Emission Line Properties
• First study of reionization in tilted flat and untilted non-flat dynamical dark energy inflation models
• The Point Spread Function Reconstruction by Using Moffatlets - I
• Karhunen-Loève Data Imputation in High Contrast Imaging
• A new off-point-less observing method for millimeter and submillimeter spectroscopy with a frequency-modulating local oscillator (FMLO)
• Determination of astrophysical parameters of quasars within the Gaia mission
• Modeling the Spectral Diversity of Quasars in the Sixteenth Data Release from the Sloan Digital Sky Survey
• Post-processing CHARIS integral field spectrograph data with PyKLIP
• A metric space for type Ia supernova spectra: a new method to assess explosion scenarios
• De-noising the galaxies in the Hubble XDF with EMPCA
• Characterization of Supernovae Based on the Spectral-Temporal Energy Distribution: Possible two SN Ib Subtypes
• Efficient Identification of Broad Absorption Line Quasars using Dimensionality Reduction and Machine Learning
• Data-Space Validation of High-Dimensional Models by Comparing Sample Quantiles
• Redshift determination through weighted phase correlation: a linearithmic implementation
• Functorial Manifold Learning
• Spectral cube extraction for the VLT/SPHERE IFS: Open-source pipeline with full forward modeling and improved sensitivity
• Functional Data Analysis for Extracting the Intrinsic Dimensionality of Spectra: Application to Chemical Homogeneity in the Open Cluster M67
• A large sub-Neptune transiting the thick-disk M4V TOI-2406
• Physical-Layer Machine Learning with Multimode Interferometric Photon Counting