Technical Program

Paper Detail

Paper Title Tightening Mutual Information Based Bounds on Generalization Error
Paper IdentifierMO4.R5.3
Authors Yuheng Bu, University of Illinois at Urbana Champaign, United States; Shaofeng Zou, University at Buffalo, The State University of New York, United States; Venugopal V. Veeravalli, University of Illinois at Urbana Champaign, United States
Session Generalization Bounds
Location Saint Victor, Level 3
Session Time Monday, 08 July, 16:40 - 18:00
Presentation Time Monday, 08 July, 17:20 - 17:40
Manuscript  Click here to download the manuscript
Abstract A mutual information based upper bound on the generalization error of a supervised learning algorithm is derived in this paper. The bound is constructed in terms of the mutual information between each individual training sample and the output of the learning algorithm, which requires weaker conditions on the loss function, but provides a tighter characterization of the generalization error than existing studies. Examples are further provided to demonstrate that the bound derived in this paper is tighter, and has a broader range of applicability. Application to noisy and iterative algorithms, e.g., stochastic gradient Langevin dynamics (SGLD), is also studied, where the constructed bound provides a tighter characterization of the generalization error than existing results.