Do any of these sound like you?

1. You’ve heard of multi-GPU training... but haven’t touched it.

2. You’re not sure what “data parallelism” or “pipeline parallelism” actually mean, let alone when to use them.

3. You’ve dabbled with DDP, but it felt more like trial-and-error than engineering.

4. You want to train larger models more efficiently. But you aren’t sure how to do it right.

5. You’ve seen mentions of ZeRO, FSDP, DeepSpeed, but don’t know how they differ or why they matter.

6. You worry that training at scale is only for people at FAANG or OpenAI.

If any of that speaks to you… this course is for you!

This is a hands-on, code-first course designed for engineers and researchers who want to understand how large-scale training actually works and be able to do it themselves. You'll go from:

"I've never used more than one GPU before, ever"

to

"I know how I could scale a training job across 2, 8, 64, or even 256 GPUs, and I understand the best ways of going about it"

If you're ready to stop watching from the sidelines and start building at scale, you belong in this cohort.

---

WHAT YOU CAN EXPECT

This course will provide you with hands-on experience training models at scale. We will meet once to twice a week for five weeks, with generous office hours and guest speakers (read below for the course schedule)

I'll also be holding office hours and host a Discord community where you can communicate with myself and other students. In return, you will learn the skills needed to keep up with today's training practices in the Deep Learning world and stay on top of the competition. All sessions will be recorded and available to students asynchronously.

---

FREE COMPUTE

To make sure all students have appropriate access to resources throughout the course to train models, Modal is sponsoring each student with $500 in compute credits towards their platform.

On top of this, Hugging Face is sponsoring each student with 6 months of Pro!

---

COURSE CONTENT

Lesson 1: Fundamentals & Distributed Data Parallelism from Scratch

- Understanding how to launch distributed jobs through Jupyter Notebooks interactively, to help bootstrap both the learning experience and make debugging smoother

- How distributed training differs when we migrate off a single GPU

- What are the best practices for making sure data is being trained on as efficiently as possible

- How does `torch.distributed.DistributedDataParallel` work and how can we build it ourselves

Lesson 2: The Zero Redundancy Optimizer (Part 1)

- Getting a grasp on what "ZeRO"/Fully Sharded Data Parallelism is

- Implementing the first stage and second stage from scratch

- Understanding how to benchmark distributed code to get accurate measurements

Lesson 3: The Zero Redundancy Optimizer (Part 2)

- Understand how the third and final stage of ZeRO is implemented

- Learn when to use each technique appropriately

- How to bootstrap each technique based on compute availability to speed up training

Lesson 4: Other Distributed Strategies and Distributed Inference

- Learn and implement Pipeline Parallelism and Tensor Parallelism from scratch

- Understand how these techniques help enable deployment of huge models at scale easily

Lesson 5: Combining Distributed Strategies

- Understand how to go from 1D parallelism to 3D parallelism

- Learn which frameworks help implement this the best

Guest Speakers:

* Sylvain Gugger (Jane Street): Topic TBD

* Marc Sun (Hugging Face): Distributed Inference

* Matej Sirovatka (Hugging Face): Expert Parallelism

* Less Wright (Meta): Async Tensor Parallelism

* Ferdinand Mom (Hugging Face): Combining DP, TP, and PP