Simpler PAC-Bayesian Bounds for Hostile Data

Pierre Alquier (1, 2) , Benjamin Guedj (3)

1 Université Paris-Saclay
2 CREST - Centre de Recherche en Économie et Statistique
3 MODAL - MOdel for Data Analysis and Learning

PAC-Bayesian learning bounds are of the utmost interest to the learning community. Their role is to connect the generalization ability of an aggregation distribution $\rho$ to its empirical risk and to its Kullback-Leibler divergence with respect to some prior distribution $\pi$. Unfortunately, most of the available bounds typically rely on heavy assumptions such as boundedness and independence of the observations. This paper aims at relaxing these constraints and provides PAC-Bayesian learning bounds that hold for dependent, heavy-tailed observations (hereafter referred to as \emph{hostile data}). In these bounds the Kullack-Leibler divergence is replaced with a general version of Csisz\'ar's $f$-divergence. We prove a general PAC-Bayesian bound, and show how to use it in various hostile settings.

Domaines: Machine Learning [stat.ML]; Théorie [stat.TH]; Statistiques [math.ST]

Pierre Alquier, Benjamin Guedj. Simpler PAC-Bayesian Bounds for Hostile Data. Machine Learning, 2018, ⟨10.1007/s10994-017-5690-0⟩. ⟨hal-01385064v2⟩