Transformers
Transformers are a short name for neural networks with self-attention, multiple heads and several layers. Over the past several years, Transformer-based models have become a central architecture in modern AI. Although Transformers first became widely known through natural language processing, they are now used across many data modalities: ChatGPT and Google Translate for text; DINO and Segment Anything (SAM) for images; Whisper for speech recognition; Sora and VideoPoet for video generation; Stable Audio for audio generation, to name a few.
In the lectures, we will study the core Transformer architecture, its most important variants and applications, and practical considerations related to training and using Transformer models. The practical part of the course (labs and project) focuses on giving you hands-on experience with Transformer models.
There will be 11 tests, labs and a course project. For the project, you are free to focus on the type of data that interests you most, such as text, speech, images, video, time series, or multimodal data.
At a glance
- Lectures: prerecorded and released on Mondays
- Tests: selected Mondays, 12:15–14:00, room 1019, Narva mnt 18
- Labs: selected Thursdays, 14:15–16:00, room 1019, Narva mnt 18
- Project: mainly from the second half of November through January
- Communication: Slack
- Assessment: tests 50 points + course project 50 points
The course starts on August 31, 2026, with the release of the first prerecorded lecture. The first in-class session will take place on Thursday, September 3, 14:15–16:00.
How the course works
During the first part of the course, we follow a cycle of lectures, tests, discussions, and practical work.
Prerecorded lectures are released on Mondays. A test covers material from the preceding lecture(s). After each test, we discuss the questions and solutions together.
Labs are held on selected Thursdays. They are not graded, but they are designed to give you practical experience with the material and prepare you for the course project.
From the second half of November, the focus shifts from regular tests and labs to the course project. You will work on a Transformer-related problem of your choice and take part in intermediate presentations and feedback sessions. The final project defence will take place at the end of January 2027.
Tests, labs, and project activities are not held every week. Please follow the detailed schedule for the exact dates. More detailed timetables and requirements are available on the following pages: tests, labs and project.
Course Prerequisites
The practical part of the course assumes that you are comfortable with:
- Python programming
- Working with Jupyter notebooks (e.g., Jupyter Notebook or Google Colab)
- Basic machine learning concepts
- Fundamentals of neural networks and deep learning
We also recommend reviewing basic concepts from:
- Probability and statistics
- Calculus
- Linear algebra
Useful refresher materials are available on the links page.
Previous experience with Transformer models is not required.
Communication
We use Slack for course discussions, questions, and announcements. All registered students will be added to the Slack workspace by the instructors, so please make sure to confirm your enrollment.
You can also join the workspace yourself using this invitation link.
When joining the Slack workspace, please use your student email address and your real full name so that we can verify your enrollment. Accounts that do not meet both of these requirements may be removed from the workspace.
For general course questions, please use the #questions channel so that everyone can benefit from the answer.
For private questions, you can contact any of the instructors directly via Slack.
For study-related administrative matters, such as illness or absence, please contact Lisa Yankovskaya via Slack or email.
Grading
The course is graded out of 100 points:
- 10 regular tests: 4 points each, for a total of 40 points
- Final test: 10 points
- Course project: 50 points
You need at least 51 points in total to pass the course. We follow the standard grading scale:
- A: 91–100 points
- B: 81–90 points
- C: 71–80 points
- D: 61–70 points
- E: 51–60 points
- F: 0–50 points
Labs do not give points, but we strongly encourage you to complete them, as they are designed to prepare you for the course project and make the project work easier.
There are no formal bonus points. However, active participation in after-test discussions and completion of the labs may be taken into account if your final score is close to the boundary between two grades.
Active participation in the course may be helpful if you are interested in writing your thesis in our group.
AI Usage
AI tools are welcome, except during tests.
Use them as a learning aid, not as a substitute for understanding. Use AI critically: verify generated answers, make sure you understand the results, and be prepared to explain any work you submit.
Instructors:
- Mark Fishel: lectures and tests
- Giacomo Magnifico: labs and tests
- Maksym Del: tests and labs
- Lisa Yankovskaya: tests, labs and course administration.
