XIII. Elements of Statistical Learning Theory
Given by Sven Laur
Brief summary: Bias-variance dilemma revisited. Training and testing data as iid samples from the distribution of future challenges. Confidence bounds on cost estimations via Monte-Carlo integration. Why is does the training error underestimate future costs. Case study for the finite function set. Bias in training error and its connection to union-bound and multiple hypothesis testing. Consistency and identifiability properties. VC-dimension as a way to estimate bias in training error. Rademacher complexity and its connection to the bias in the training error. Limitations of statistical learning theory.
Slides: PDF
Video: UTTV (2016) UTTV (2015) UTTV (2014)
Literature:
- Cristianini & Shawe-Taylor: Support Vector Machines: Generalisation Theory (Chapter 4)
- Bartlett & Mendelson: Rademacher and Gaussian Complexities: Risk Bounds and Structural Results
- David MacKay: Information Theory, Inference, and Learning Algorithms: Capacity of a Single Neuron
Complementary exercises:
- Estimate the difference between training and test errors for different classifiers
- Draw data from linearly separable model with some gaussian random shifts
- Try various linear and non-linear classifiers
- Plot the discrepancy as a function of training sample size
- Draw data form more complex model that cannot be represented by predictor classes
- Repeat the procedure
- Estimate VC and Rademacher complexities and see if SLT bounds coincide with practice
- Estimate the difference between training and test errors for different prediction algorithms
- Draw the data form a linear model
- Try various linear and non-linear predictors
- Plot the discrepancy as a function of training sample size
- Draw data form more complex model that cannot be represented by predictor classes
- Repeat the procedure
- Estimate VC and Rademacher complexities and see if SLT bounds coincide with practice
Free implementations:
- None that we know