Masinõpe II 2018/19 kevad

I. Performance evaluation measures

Given by Sven Laur

Brief summary: Theoretical formalisation of performance measures. Loss function and risk. Empirical risk and its convergence to risk. Confidence intervals for risk estimates and the size of the hold-out test set. Why training error is biased. Optimism as a way to correct training error. Bias-variance dilemma and crossvalidation as an engineering solution. Properties of k-fold crossvalidation. Moment matching and naive confidence intervals for the crossvalidation error. Why the naive estimate underestimates the variance of the crossvalidation error. Bootstrap as an alternative to crossvalidation.

Literature

• Bousquet, Boucheron and Lugosi: Introduction to Statistical Learning Theory
• Bengio and Grandvalet: No Unbiased Estimator of the Variance of K-Fold Cross-Validation
• Jüri Lember:Tehisõppe loengukonspekt
• Friedman, Tibshirani and Hastie: The Elements of Statistical Learning