Suboptimal behaviour of Bayes and MDL in classification under misspecification
- Peter Grunwald ,
- John Langford
Computational Learning Theory 2004 |
This is a slightly longer version of the paper at COLT 2004, containing tow extra pages of discussion of the main results.
We show that forms of Bayesian and MDL inference that are often applied to classification problems can be *inconsistent*. This means there exists a learning problem such that for all amounts of data the generalization errors of the MDL classifier and the Bayes classifier relative to the Bayesian posterior both remain bounded away from the smallest achievable generalization error.