# Hands on Deep Learning

Deep learning is a trending topic in machine learning community. In the last decade this field has made a **giant step** forward and the achievements are quite astonishing:

- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Deepmind artificial intelligence @ FDOT14
- The wonderful and terrifying implications of computers that can learn @ TED
- Computers Teach Themselves to Recognize Cats, Faces
- Speech Recognition Breakthrough for the Spoken, Translated Word

In the last semester **we learned the basics** of deep learning from the Hintonâ€™s Coursera course. This seminar is a follow-up, where we will **apply in practice** what we have learned.

You still can take the seminar **even if you haven't participated** in the previous one.

## Organization

In the course of semester we will do two things:

Implement a simple algorithm on a toy example ^{(weeks 1 - 6)}

Each student will **pick a narrow topic** from the area of deep learning (for example: "conjugate gradient descent" or "Nesterov momentum"), implement the algorithm and validate it on a toy example. The aim here is to understand the selected topic in-depth and, once implemented, explain it to fellow students. You can come up with the topic yourself or choose from the list:

- L1 vs. L2 regularization in artificial neural networks
- Conjugate gradient descent vs. usual gradient descent
- RMSProp vs. usual gradient descent
- Use a single neural network architecture, but average predictions made by many different sets of weights, that predict the output (Lecture 9 Video 1 @ 2:48)
- Penalize subset of weights in a network (Lecture 9 Video 2)
- Stochastically binarize outputs of activation functions based of their activity during the forward pass. Get the error. Do backpropagation as usual. (Lecture 9)
- Implement toy example of Full Bayesian approach (Lecture 10 Video 3 @ 7:20)
- Instead of Full Bayesian do Monte Carlo, run it 1000 times and confirm that it approximates the full computation and then increase the size of the network to see how much you can achieve (Lecture 10 Video 4)
- Backpropagation on GPU (CUDA) vs. CPU
- First 2 homeworks from the Stanford's Convolutional Neural Networks course.
- Backpropagation in R or any other exotic language.
- Optimisation in word2vec
- ADAgrad, ADAdelta optimisation algorithms
- Levenbergâ€“Marquardt algorithm: Damped Leased Squares
- Rectangular Convolutional Filter

Apply your knowledge on a real-world dataset ^{(week 7 - 16)}

Working alone or in group, you will **apply deep learning to a real-world dataset** from your master or PhD research project, Kaggle competition or industry and report the results.

## Format

The seminar will happen in the format of a **hackathon**: students will meet, develop and discuss arising problems. There will be several **keypoint seminars** in the middle and at the end of the semester, where everyone should be present and demonstrate their results.

## Administrative details

Date and location: **Wed 14:15 @ Liivi 2-512**

Contact:

To pass the course you need to

- study one algorithm in detail and explain it to the others
- apply deep learning to a real-world problem and present the results

## References

Fall Semeter 2014 https://courses.cs.ut.ee/2014/dmseminar/fall/Main/DeepMining