This started as a distributed training course. It organically grew into an all-encompassing learning event with world-class speakers who have deep knowledge and experience when it comes to the modern techniques used to train at scale today. The distributed training course is still here, in its full capacity.

But now during each week there are hand-tailored speakers discussing topics directly relating to the tips and techniques brought up in that weeks lecture

to help enhance your understanding of concepts and modern practices to completion.

All materials and recordings will be available to participants who enroll. There are 14 talks across three tracks and 5 lessons (and growing) in addition to office hours.

Conference Talks

------------------

Applied Track

------------------

Hear from industry leaders how they have approached the problems of training at scale, and the solutions they've made

Robert Nishihara: Cofounder of Ray, Anyscale

- How Ray and Anyscale help you scale training across thousands of GPUs

Sami Jaghouar: Research Engineer, Prime Intellect

- How Decentralized training at a global scale is possible, and how Prime Intellect gets it done

Tunji Ruwase: Software Engineer, Snowflake, formerly Microsoft

- How Arctic Long Sequence Training makes training multi-million token context length efficient and scalable

Prince Canuma: Machine Learning Research Engineer

- How MLX (Apple Silicon) lets you combine M-series Macs to run machine learning workloads locally for a fraction of the cloud cost

Pretraining Track:

------------------

Pretraining is a core foundation of modern LLM research. Learn what techniques are used today for creating the best model possible

Phuc Nguyen: Research Engineer, Hugging Face

- A Practitioners Guide to FP8 Training

Elie Bakouch: Machine Learning Researcher, Hugging Face

- How modern LLM's like DeepSeek and others are hyper-optimized for efficient training through techniques like MLA, MoE, and more

Daniel Han: Creator of UnslothAI, formerly NVIDIA

- How Triton kernels and other techniques save you hundreds of hours in training time

Distributed Training Course

------------------

Learn the techniques used today when training and fine-tuning models at scale (hundreds and thousands of GPUs at once). Five workshops

help guide you through implementing the most common techniques from the ground up, including Distributed Data Parallelism, the entirety

of ZeRO, and more.

Workshop 1: Distributed Data Parallelism from scratch, and how to make sure your data isn't your bottleneck

Workshop 2: The Zero Redundancy Optimizer (Part 1): How model sharding helps train large models across smaller, multiple GPUs at once

Workshop 3: The Zero Redunancy Optimizer (Part 2): How the different levels of ZeRO effect training time, and which are the most efficient for your use case

Workshop 4: Pipeline and Tensor Parallelism: How and why these techniques help solve the biggest slowdown when training: communication

Workshop 5: Multi-Dimensional Parallelism: Why today we combined all of the techniques above at once to get the highest throughput possible during training

The Distributed Training Course has these guest lecturers and topics:

Sylvain Gugger: Jane Street, formerly Hugging Face & fast.ai

- Introduction to Distributed Training, and an overview of ZeRO

Wanchao Liang: Formerly Meta, Creator of TorchTitan/DTensor

- How TorchTitan has helped developers take model pretraining to scale faster, and how DTensors have made this easier

Ferdinand Mom: Research Engineer, Hugging Face

- What is multi-dimensional parallelism, and how this is a crucial technique to training models at scale today

Less Wright: PyTorch Partner Engineer, Meta

- How Async TensorParallelism helps you train at scale efficiently

Matej Sirovatka: Machine Learning Engineer, Hugging Face

- What is Expert Parallelism and why it's needed when training Mixture-of-Expert models

Marc Sun: Machine Learning Engineer, Hugging Face

- Tips and tricks needed for deploying large models at scale

Free Compute

------------------

To help give you hands on experience with training models, we're sponsored by the following companies to help get you working on training models at scale from Day 1:

- Hugging Face: 6 months of Pro

- Modal: $500 in compute credits

More to be announced