Master distributed training and real-world scale techniques from top engineers.

Whether you're an ML engineer looking to move beyond single-GPU experiments, or a product leader seeking to understand the language your AI team speaks, this course will give you the hands-on skills and conceptual clarity to operate confidently at scale.

What Makes This Course Different

Rather than saying "here's PyTorch FSDP, it works like such, and now use this configuration" we're going to build every core parallelism strategy from scratch.

I'm not here to tell you how to use a framework. I'm here to tell you how to ensure that your implementations are sound, you can ace that interview, and you know just what the tools you're using are up to for large-scale training.

All course materials and recordings are available after enrollment, and you’ll get free lifetime access to future cohorts.

💡 What’s included:

6+ core lessons split across 4 weeks.

3+ workshops with hands-on experience with tricks I've personally learned, used, and am actively developing

Also will include all 14+ guest lectures from the prior cohort

$2000 in compute from Modal and Lambda

Class discord with lifetime access

100% money-back guarantee (within 14 days of starting the course)

🧠 Distributed Training Course

Learn the foundations and modern techniques used in real-world LLM scaleups across hundreds or thousands of GPUs.

5 core instructor-led workshops:

DDP from scratch and avoiding data bottlenecks

ZeRO (Part 1): How model sharding enables scale

ZeRO (Part 2): Efficiency tradeoffs and stage comparison

Pipeline & Tensor Parallelism: Solving communication slowdowns

Multi-Dimensional Parallelism (recorded from last cohort): Combining all methods for throughput

Plus other targeted workshops covering:

* How DataLoaders work with distributed training

* Using FP8 in the real world (and on consumer hardware)

* How PyTorch traces help us verify our implementations

{More planned/on the way}

Prior (included) cohort talks include:

🏕️ Fireside Chats

Hear from the experts on their real experiences trying to take models to scale, the challenges, and the discoveries

Yuxiang Wei (Meta FAIR)

📣 Conference Talks

Applied Track

Hear how industry leaders are solving real-world scale problems:

Robert Nishihara (Ray, Anyscale): Scaling across thousands of GPUs with Ray

Sami Jaghouar (Prime Intellect): Decentralized global-scale training

Tunji Ruwase (Snowflake): Efficient long-context training with Arctic

Prince Canuma: Local ML workloads using Apple Silicon + MLX

Pretraining Track

Deep dives into LLM pretraining at scale:

Phuc Nguyen (Hugging Face): A practitioner's guide to FP8

Elie Bakouch (Hugging Face): Hyper-optimizing LLMs with MoE, MLA & more

Daniel Han (UnslothAI): Speeding up training with Triton & custom kernels

Guest Lectures include:

Sylvain Gugger (Jane Street): Overview of ZeRO

Wanchao Liang (TorchTitan): DTensor and large-scale pretraining

Wing Lian (Axolotl): 2D Parallelism with Axolotl

Ferdinand Mom (Hugging Face): Multi-dimensional parallelism

Less Wright (Meta): Async TensorParallelism

Matej Sirovatka (Hugging Face): Expert Parallelism for MoE

Marc Sun (Hugging Face): Deployment strategies at scale

✅ Guarantee

If you're not satisfied, we offer a 100% refund up to 14 days after the course begins. No risk, just learning.