LTAT.06.030 Parallelism in Deep Learning
Course Overview
This course focuses on parallelism in deep learning systems, with an emphasis on techniques used to scale training and inference on modern GPU hardware. Students will build on existing deep learning knowledge to understand how computation, memory, and communication interact in large models. Using Python tools such as NumPy and PyTorch, the course explores data parallelism, model parallelism, pipeline parallelism, and hybrid strategies as applied in real-world systems. By the end of the course, students will be able to analyze performance bottlenecks, design parallel training strategies, and implement efficient multi-GPU deep learning workflows.
Course Info
- Lectures: TUE 14:15 Delta 1004 (weeks 1-16)
- Labs: THR 16:15 Delta 2047 (weeks 1-16)
- Instructor: Mohammad Anagreh, Delta 3043, (mohammad.anagreh@ut.ee)
- Credits: 6 ECTS
- Language: English
- Lectures: Pre-recorded video lectures
- Tools: PyTorch (no low-level CUDA/OpenCL programming required)
- Slack channel:
Structure
The course is organized into lectures and practical sessions, progressing from theory to hands-on implementation:
- Lectures
- Part 1: Foundations of Parallelism & Deep Learning (Weeks 1–4)
- Part 2: Core Parallel Strategies in Practice (Weeks 5–11)
- Part 3: Project Work and Assessment (Weeks 12–16)
- Practical Sessions:
- Each lecture is paired with hands-on labs where students implement the discussed strategies in Python and PyTorch, applying parallelism techniques on GPUs.
Grading
- Homeworks: 60%
- Homework 1: 15%
- Homework 2: 15%
- Homework 3: 15%
- Homework 4: 15%
- Project: 40%
- Total: (Homeworks) 60% + (Project) 40% = 100%