Neural Network Training Dynamics
This course is based on materials from the "Topics in Machine Learning: Neural Net Training Dynamics" course developed by Roger Grosse and available at https://www.cs.toronto.edu/~rgrosse/courses/csc2541_2021/.
This class is about developing the conceptual tools to understand what happens when a neural net trains. Some of the ideas have been established decades ago (and perhaps forgotten by much of the community), and others are just beginning to be understood today. Here are some of the topics that we plan to cover during the course, although the schedule may likely change as the course goes on:
- Linear Regression
- Taylor Approximations
- Metrics
- Second-Order Optimization
- Adaptive Gradient Methods, Normalization, and Weight Decay
- Infinite Limits and Overparameterization
- Stochastic Optimization and Scaling
- Bayesian Inference and Implicit Regularization
- Dynamical Systems and Momentum
- Bilevel Optimization
Prerequisites
Hard requirements for this course are Calculus and Machine Learning. It is recommended that you are also comfortable with either Neural Networks or Optimization Methods.
This is going to be a tough and mathy course, so please do not take the prerequisites lightly.
Organization
Lectures will be hybrid, meaning both online and offline. The class time is Fridays 12:15-14:00 in Delta room 2047 and over Zoom (link here. The password is ati).
We will be using Slack for communication between students and instructors. If you are registered to the course, you should have received an invitation link to the #course-nntd Slack channel. In case you haven't, for some reason, please contact: victor.pinheiro at ut.ee
Grading
To pass the course, students need to:
- submit all three homeworks and score at least 60% of average points; (grades sheet available here.)
- either present one homework or present a lecture;
- attend 80% of the lectures. (check your attendances here.)
Contacts
Tambet Matiisen (tambet.matiisen at ut.ee) - coordinating the course.
Victor Pinheiro (victor.pinheiro at ut.ee) - teaching assistant.