Toward Optimal Feature Selection in Naive Bayes for Text Categorization

Abstract

Automated feature selection is important for text categorization to reduce the feature size and to speed up the learning process of classifiers. In this paper, we present a novel and efficient feature selection framework based on the Information Theory, which aims to rank the features with their discriminative capacity for classification. We first revisit two information measures: Kullback-Leibler divergence and Jeffreys divergence for binary hypothesis testing, and analyze their asymptotic properties relating to type I and type II errors of a Bayesian classifier. We then introduce a new divergence measure, called Jeffreys-Multi-Hypothesis (JMH) divergence, to measure multi-distribution divergence for multi-class classification. Based on the JMH-divergence, we develop two efficient feature selection methods, termed maximum discrimination ($MD$) and $MD-{\chi }^2$ methods, for text categorization. The promising results of extensive experiments demonstrate the effectiveness of the proposed approaches.

This paper has been submitted to the IEEE Trans. Knowledge and Data Engineering. 14 pages, 5 figures

References in corpus (3)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Sentences and Documents
• EEF: Exponentially Embedded Families with Class-Specific Features for Classification

Cited by in corpus (9)

• Variable selection for Naïve Bayes classification
• Short text classification with machine learning in the social sciences: The case of climate change on Twitter
• A Semi-Supervised Adaptive Discriminative Discretization Method Improving Discrimination Power of Regularized Naive Bayes
• EEF: Exponentially Embedded Families with Class-Specific Features for Classification
• Boosting the Discriminant Power of Naive Bayes
• Pretzel: Email encryption and provider-supplied functions are compatible
• Unified Likelihood Ratio Estimation for High- to Zero-frequency N-grams
• FSMJ: Feature Selection with Maximum Jensen-Shannon Divergence for Text Categorization
• Kernel-based Generative Learning in Distortion Feature Space