Feature selection with Rényi min-entropy

Catuscia Palamidessi (1) , Marco Romanelli (2, 3, 1)

1 COMETE - Concurrency, Mobility and Transactions
2 UNISI - Università degli Studi di Siena = University of Siena
3 X - École polytechnique

We consider the problem of feature selection, and we propose a new information-theoretic algorithm for ordering the features according to their relevance for classification. The novelty of our proposal consists in adopting Rényi min-entropy instead of the commonly used Shannon entropy. In particular, we adopt a notion of conditional min-entropy that has been recently proposed in the field of security and privacy, and that avoids the anomalies of previously-attempted definitions. This notion is strictly related to the Bayes error, which is a promising property for achieving accuracy in the classification. We evaluate our method on 2 classifiers and 3 datasets, and we show that it compares favorably with the corresponding one based on Shannon entropy.

Domaines: Apprentissage [cs.LG]; Théorie de l'information [cs.IT]; Informatique [cs]

Catuscia Palamidessi, Marco Romanelli. Feature selection with Rényi min-entropy. Artificial Neural Networks in Pattern Recognition - 8th IAPR TC3 Workshop (ANNPR 2018), Sep 2018, Siena, Italy. pp.226-239. ⟨hal-01830177v2⟩