Course syllabus:
Part 1: Foundations of Parallelism & Deep Learning (Weeks 1–4)
- Week 1: Introduction & Mathematical Refresher
- Week 2: Hardware & Parallelism Basics
- Week 3: Profiling & Bottlenecks
- Week 4: Overview of Parallelism Strategies
Part 2: Core Parallel Strategies in Practice (Weeks 5–11)
- Week 5: Data Parallelism with DataParallel (DP)
- Week 6: DistributedDataParallel (DDP) & Collective Communication
- Week 7: DDP Optimization and Debugging
- Week 8: Model Parallelism (Layer-wise)
- Week 9: Pipeline Parallelism
- Week 10: Hybrid Parallelism
- Week 11: Recap and Project Q&A
Part 3: Project Work and Assessment (Weeks 12–16)
- Week 12: Project Work Session 1
- Week 13: Project Work Session 2
- Week 14: Project Work Session 3
- Week 15: Final Project Presentations
- Week 16: Assessment