Fisher and Kernel Fisher Discriminant Analysis: Tutorial

Abstract

This is a detailed tutorial paper which explains the Fisher discriminant Analysis (FDA) and kernel FDA. We start with projection and reconstruction. Then, one- and multi-dimensional FDA subspaces are covered. Scatters in two- and then multi-classes are explained in FDA. Then, we discuss on the rank of the scatters and the dimensionality of the subspace. A real-life example is also provided for interpreting FDA. Then, possible singularity of the scatter is discussed to introduce robust FDA. PCA and FDA directions are also compared. We also prove that FDA and linear discriminant analysis are equivalent. Fisher forest is also introduced as an ensemble of fisher subspaces useful for handling data with different features and dimensionality. Afterwards, kernel FDA is explained for both one- and multi-dimensional subspaces with both two- and multi-classes. Finally, some simulations are performed on AT&T face dataset to illustrate FDA and compare it with PCA.

arXiv commentv2: updated some sections, corrected some typos, and applied readers' feedback

References in corpus (6)

• Linear and Quadratic Discriminant Analysis: Tutorial
• The Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial
• Eigenvalue and Generalized Eigenvalue Problems: Tutorial
• Feature Selection and Feature Extraction in Pattern Analysis: A Literature Review
• Recognizing Involuntary Actions from 3D Skeleton Data Using Body States
• Unsupervised and Supervised Principal Component Analysis: Tutorial

Cited by in corpus (12)

• Fisher Discriminant Triplet and Contrastive Losses for Training Siamese Networks
• Locally Linear Embedding and its Variants: Tutorial and Survey
• Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nyström Method, and Use of Kernels in Machine Learning: Tutorial and Survey
• Sampling Algorithms, from Survey Sampling to Monte Carlo Methods: Tutorial and Literature Review
• Quantile-Quantile Embedding for Distribution Transformation and Manifold Embedding with Ability to Choose the Embedding Distribution
• Laplacian-Based Dimensionality Reduction Including Spectral Clustering, Laplacian Eigenmap, Locality Preserving Projection, Graph Embedding, and Diffusion Map: Tutorial and Survey
• Johnson-Lindenstrauss Lemma, Linear and Nonlinear Random Projections, Random Fourier Features, and Random Kitchen Sinks: Tutorial and Survey
• Quantized Fisher Discriminant Analysis
• Meta-learning One-class Classifiers with Eigenvalue Solvers for Supervised Anomaly Detection
• Regularized Deep Linear Discriminant Analysis
• Weighted Fisher Discriminant Analysis in the Input and Feature Spaces
• Roweisposes, Including Eigenposes, Supervised Eigenposes, and Fisherposes, for 3D Action Recognition