Every training framework wraps around the same core distributed training strategies:

- torch's Distributed Data Parallelism

- DeepSpeed

- torch's FullyShardedDataParallelism

However, APIs can break. Strong deprecations in the name of "clean code" can destroy your entire stack.

This course is designed to help you create your own scalable training framework by incorporating the most common tools in the industry.

This is not a "how does DDP work" course.

This is a "how do I incorporate DDP into an internal training framework that is stable and works well"

Our aim is at the end of 3 weeks to create a minimal version of Hugging Face's Accelerate with a few different spins.

This includes:

1. Writing the source code

2. Learning how to test distributed code well

3. How do you create CI's in Github for distributed testing

I'll give you all the tricks I've learned over 4 years in developing accelerate into one class to help you not only write frameworks yourself but also be able to understand common patterns in other frameworks to make navigating them easier

To help facilitate the learning, Prime Intellect will be sponsoring $300 in compute per student