Walk with Code

Cohort 1

Cohort 2

Remembering how each torch.distributed operation behaves can be difficult. Here's an easy PDF with visuals of each kind showcasing how tensors get modified throughout all of the common MPI operations

Distributed Operations Cheatsheet

The Distributed Training Lexicon is a free resource of 49 different distributed training terms with pairing definitions and some visualizations to go with it. The goal is to have a very quick cheatsheet to look at when needing a reminder of what certain methods are.

Distributed Training Lexicon

Hi there! To help you get a good grasp on how the course will be oriented and an idea on what some of the content looks like, I can share with you an exclusive preview into what the course webpage will be and how some of the content is shaped. I've worked hard to make sure Quarto and Jupyter will help me create educational material that will wow you, so let me know if it does! (Note: this material preview may change as the course develops, but only for additive purposes)

Free Access to Part of Lesson 1

Learn the techniques used today to take your model training from Colab to Clusters

Zachary Mueller

Scratch to Scale: Large-Scale Training in the Modern World

Lead Software Engineer II

Senior System Software Designer

Founder

Sr deep learning engineer

PhD Student

Quantitative Risk Analyst

Software Engineer

Master's Student

Senior Manager, Applied Science

Master distributed training and real-world scale techniques from top engineers.Whether you're an ML engineer looking to move beyond single-GPU experiments, or a product leader seeking to understand the language your AI team speaks, this course will give you the hands-on skills and conceptual clarity to operate confidently at scale. What Makes This Course DifferentRather than saying "here's PyTorch FSDP, it works like such, and now use this configuration" we're going to build every core parallelism strategy from scratch. I'm not here to tell you how to use a framework. I'm here to tell you how to ensure that your implementations are sound, you can ace that interview, and you know just what the tools you're using are up to for large-scale training. All course materials and recordings are available after enrollment, and you’ll get free lifetime access to future cohorts. 💡 What’s included: 6+ core lessons split across 4 weeks.3+ workshops with hands-on experience with tricks I've personally learned, used, and am actively developingAlso will include all 14+ guest lectures from the prior cohort$2000 in compute from Modal and LambdaClass discord with lifetime access100% money-back guarantee (within 14 days of starting the course) 🧠 Distributed Training CourseLearn the foundations and modern techniques used in real-world LLM scaleups across hundreds or thousands of GPUs.5 core instructor-led workshops:DDP from scratch and avoiding data bottlenecksZeRO (Part 1): How model sharding enables scaleZeRO (Part 2): Efficiency tradeoffs and stage comparisonPipeline &amp; Tensor Parallelism: Solving communication slowdownsMulti-Dimensional Parallelism (recorded from last cohort): Combining all methods for throughput Plus other targeted workshops covering:* How DataLoaders work with distributed training* Using FP8 in the real world (and on consumer hardware)* How PyTorch traces help us verify our implementations{More planned/on the way} Prior (included) cohort talks include:🏕️ Fireside Chats Hear from the experts on their real experiences trying to take models to scale, the challenges, and the discoveries Yuxiang Wei (Meta FAIR) 📣 Conference Talks Applied TrackHear how industry leaders are solving real-world scale problems: Robert Nishihara (Ray, Anyscale): Scaling across thousands of GPUs with RaySami Jaghouar (Prime Intellect): Decentralized global-scale trainingTunji Ruwase (Snowflake): Efficient long-context training with ArcticPrince Canuma: Local ML workloads using Apple Silicon + MLX Pretraining TrackDeep dives into LLM pretraining at scale: Phuc Nguyen (Hugging Face): A practitioner's guide to FP8Elie Bakouch (Hugging Face): Hyper-optimizing LLMs with MoE, MLA &amp; moreDaniel Han (UnslothAI): Speeding up training with Triton &amp; custom kernels Guest Lectures include:Sylvain Gugger (Jane Street): Overview of ZeROWanchao Liang (TorchTitan): DTensor and large-scale pretrainingWing Lian (Axolotl): 2D Parallelism with AxolotlFerdinand Mom (Hugging Face): Multi-dimensional parallelismLess Wright (Meta): Async TensorParallelismMatej Sirovatka (Hugging Face): Expert Parallelism for MoEMarc Sun (Hugging Face): Deployment strategies at scale ✅ GuaranteeIf you're not satisfied, we offer a 100% refund up to 14 days after the course begins. No risk, just learning.

Build the skills to answer the call when it's time to take your models to scale

Course syllabus

I don't want you to be an expert. But you should have some mild experience training a model of some capacity, be it PyTorch, TF, and such

Train any model, at least once

I'm not here to teach you matrix calculus, nor will we be going that advanced. However some amount of core math is still needed

Understand basic high-school algebra

PyTorch is in Python, so having some experience with the language will do you well since the whole course is in it

Familiarity with Python coding

Tuesdays & Thursdays

Feel free to type out dates as your title as a way to communicate information about specific live sessions or other events.

May 7, 2022

Schedule items can also be used to convey commitments outside of specific time slots (like weekly projects or daily office hours).

Weekly projects

Course schedule

Beginner to intermediate MLE’s wanting to make sure they have skills that are relevant in today’s market

Senior engineers tired of piecing together half-solutions from publications, frameworks, and more.

Team leads who want confidence that engineers can execute at scale without burning time.

CTOs who need to make fast, informed decisions on how to scale LLMs.

Built for the People Scaling What’s Next

What You'll Achieve

You’ll understand the core problems teams face during large-scale training and how to avoid them using proven methods.

Train 100B+ models across 8–1,000 GPUs efficiently

You won’t just watch; you’ll train models using DDP, ZeRO, pipeline parallelism, and more. Each one applied in code.

Build real-world experience with modern training techniques

You’ll learn how to match technique to context. Whether it’s model size, hardware limits, or team constraints, you’ll know what fits.

Understand which training methods to use and when

Most teams wait too long to prepare for scale. This course makes sure you’re ready before your current training setup stops working.

Be ready before training becomes your bottleneck

You’ll connect theory with practice and walk away with working knowledge you can apply in real systems.

Go from scattered tutorials to production-ready training skills

Generous office hours ensure that students can ask questions about their specific issues, interests, and needs.

Personalized Instruction

How to train your tiny MoE: Going from 60+ hours to 13

Zach Mueller

Technical Lead, Hugging Face

What are the first levers to pull when trying to get the best performance during training

How to find the easy wins during model training

What is the torch profiler? How do we look at what's slow

How to tell *what* is causing slowdowns

How do you then find easy wins in the modeling code for increasing your FLOPs

What to do when you've optimized the training loop

How to Make High-Level Frameworks Work For You (Part 3)

Instructor, Technical Lead at Hugging Face

Is using high level frameworks bad?

If you know how it works, you know how to fix it when it breaks

Use the tools that help you get somewhere the fastest

Focus on the outcome, not the setup

Why Low-Level Training Frameworks Save Us (Part 2)

The Trainer helps you get off the ground, but is it the best way to learn by digging through thousands of lines of code?

What's the best medium to learn what's actually happening?

Since at most they're a light wrapper of a concept, low level API's make code approachable for learning

Low-level frameworks help teach you

Since they are *under abstracted*, low level frameworks help get you started faster in the long-run

Less bumps in the road

Why High-Level Training Frameworks are a Crutch (Part 1)

Having extremely high abstraction levels are great for beginners when it comes to training models, however

Helpful, at first

When you get an error, the inexperienced spend more time debugging than going

Debug hell

Hit an error? Beginners might not know what that means or why

Run and hope for the best

The Distributed Training Taxonomy: DP, PP, and more

Zach Muelller

Pipeline parallelism? Tensor parallelism? ZeRO? We'll learn at the high level what these all are

Just what are these weird names?

I'm not here to overwhelm you with code implementations and applications. This is a brief introduction to their ideas.

Focus on the idea, not the code

Figuring out the best topology of strategies is complicated. I'll help guide that decision for you

When should I use what?

How to Utilize a Cluster: All About SLURM

We will learn *why* tools like SLURM, Kubernetes, and the like exist for when people mention "scale"

Why is managing multiple computer difficult?

Why is SLURM my go-to choice between them all? What makes it special? Isn't it more... intense? (Spoiler: no)

Why SLURM?

We'll discuss why knowing SLURM (or in general how these systems work) is imperative in this modern age of Deep Learning

I just use Colab... do I really need to know this?

We're going to see how I've managed a SLURM cluster *in my house*, so that you know how to apply it to your own cluster

Learn directly from Walk with Code