A Constructive Density-Ratio Approach to Mutual Information Estimation: experiments in feature selection

Abstract

Mutual Information (MI) estimation is an important component of several data mining tasks (e.g. feature selection). In classification settings,  MI estimation essentially depends on the estimation of the ratio of two probability densities. Using a recently developed method of density-ratio estimation, which is constructive in nature, new estimators for MI can be derived. In this article, we consider one such new estimator - VMI - and compare it experimentally to previously proposed MI estimators. The first batch of experiments is conducted solely on mutual information estimation, and shows that VMI compares favorably to previous estimators. The second batch of experiments applies MI estimation to feature selection in classification tasks, evidencing that VMI leads to better feature selection performance. Combining the results of both experimental batches, we conclude that the development of improved density-ratio estimators can positively impact MI estimation and feature selection.