Hugo & Stefan

Build a Document Processing Pipeline for RAG Systems

Stitch Fix

Nextdoor

LinkedIn

Honda Research Institute

Stanford University

Stefan Krawcyk

Co-creator of Hamilton & Burr, CEO & Co-Founder DAGWorks Inc.

Understand the different components required: document loading, parsing, text chunking, and embedding creation.

Setup: the components that you need to have

Set up a basic pipeline to parse some technical documentation to get you started using open source tools.

Develop: we'll write some code

Common caveats when taking such a system to production.

Caveats: going from development to production

Become the RAG Product Leader Your Company Needs Companies are racing to integrate AI features that truly deliver value. To stay ahead, you must master the art of building Retrieval-Augmented Generation (RAG) products that continuously improve—no luck or guesswork required. 6 Weeks to Transform Your AI Product Sense This course removes the mystery and turns RAG from a risky experiment into a structured, data-driven practice. You’ll learn how to pinpoint what’s working, diagnose what’s not, and steadily raise the bar on performance and user satisfaction. What You’ll Learn, Week-by-Week Week 1: Fundamentals &amp; Synthetic DataEstablish a repeatable system for evaluating and improving RAGGenerate synthetic datasets for fast, meaningful experimentsMaster precision, recall, and essential retrieval metricsLay the groundwork for a self-reinforcing improvement cycle Week 2: Segmentation &amp; AnalysisBreak down complex query patterns and user behaviorsSpot inventory gaps and capability shortfalls quicklyUse clustering, classification, and dashboards to track progressSet improvement priorities that deliver maximum impact Week 3: Structured Data &amp; MultimodalityExtract and leverage metadata for more accurate retrievalHandle images, documents, and tables without losing qualityCreate specialized indexes and blend them effectivelyApply earlier techniques to broaden RAG capabilities Week 4: Query Routing &amp; Tool SelectionIntegrate multiple search methods into a single coherent productRoute queries intelligently using parallel function callsTurn tool selection into a quantifiable, improvable taskSecure higher success probabilities end-to-end Week 5: Representations &amp; Fine-TuningMove beyond generic embeddings into fine-tuned, domain-specific vectorsDevelop re-rankers that drastically improve retrieval qualityDiscover how small amounts of targeted data can yield big gainsShorten the experimentation cycle for continuous improvement Week 6: Product UX &amp; Feedback LoopsDesign user interfaces that collect feedback effortlesslyUse streaming and visual hints to make latency feel negligibleImplement citations, follow-up actions, and chain-of-thought promptingAdd validators and checks to ensure reliability at scale Why This MattersRAG has become a core differentiator for AI-driven products. The strategies you learn here will never go out of style. As tools and models evolve, you’ll know how to integrate them, measure their impact, and refine your approach—over and over. What’s IncludedProven evaluation frameworks that demystify “is it working?”Techniques to handle more data types, faster and more accuratelyMethods for continuous feedback collection and refinementInsights that adapt as the AI landscape changes About Your Instructor Jason Liu is a seasoned AI consultant who has guided ai engineering at Stitch Fix, and high-growth startups from Seed stage to public companies. He also advises many of today's generative AI startups. Jason specializes in transforming vague AI ambitions into concrete, measurable practices that deliver real business value. Exclusive Bonuses ($1,500+ Value)$500 Cohere credits$200 LanceDB credits + free Lance Cloud access$500 Modal Labs credits6 months of Notion AI Plus3 months of Braintrust access ($250 value) Who Should ApplyProduct teams integrating AI features into real software productsAI engineers moving from prototype to production-grade performanceAnyone committed to developing true “AI product sense”—not just hype You Only Pay After Acceptance &amp; Full Refund GuaranteeWe’re so confident in the course’s value that if you don’t see meaningful improvements in your product’s performance within five weeks of starting, we’ll refund you fully. Don’t let uncertainty stall your AI progress. Join a small cohort of dedicated practitioners, learn the proven frameworks, and build the RAG systems your users will love.

Acquire the skills & confidently improving and iterate on RAG applications

Provide a brief description for each module as a way to sell the value of what students will be learning throughout the course.

First Module

Second Module

Third Module

Course syllabus

Workshops covering each step of the playbook and helping you build process improvements in your RAG application

Tuesday: Workshops

The first half hour will be interactive breakout sessions, and the closing half-hour each week is Q&amp;A

Wednesday: Office Hours + Breakout Sessions

Guest instructors covering key topics in both innovative theory and practical applications in RAG system development.

Thursday: Guest Speakers

Course Schedule Each Week

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

This Course Is For You If You Are

By the end of this course, participants will be able to

Implement the Data and Evals Flywheel approach to continuously develop and improve RAG applications—breaking free from guesswork and relying on measurable, iterative enhancements.

Adopt a Systematic, Data-First Methodology

Quickly assess your retrieval systems using precision and recall metrics, identify bottlenecks, and confidently validate changes without sinking into endless trial-and-error cycles.

Run Fast, Unit-Test-Like Evaluations

Generate and utilize synthetic data sets to speed up experimentation, enabling you to test new approaches, embeddings, and architectures before committing full resources.

Leverage Synthetic Data for Rapid Iteration

Apply fine-tuning strategies for embedding models to boost search relevance and explore hard negative examples to further sharpen retrieval performance.

Master Fine-Tuning & Hard Negative Mining

Use query classification and segmentation techniques to pinpoint exactly where your RAG system falls short—whether it’s due to limited inventory or insufficient capabilities.

Classify Queries & Identify Bottlenecks

Create tailored indices for documents, images, tables, SQL databases, and more. Learn when and how to fuse or layer these indices to handle complex retrieval tasks elegantly.

Design Specialized Indices for Multiple Modalities

Implement synthetic text chunk generation and strategic summarization methods to improve retrieval results, ensuring end-users get clear, concise, and contextually rich answers.

Enhance Retrieval with Summarization & Chunking

Develop systems that intelligently route queries to the right indices, tools, or pipelines. Blend and fuse indices effectively to handle nuanced, multi-step queries.

Implement Query Routing & Index Fusion

Evaluate the performance of your routing logic and each individual index separately. Gain the nuance to fine-tune global system performance and local retrieval accuracy in tandem.

Optimize Both Global & Local Performance

Design explicit and implicit feedback mechanisms—capturing user signals, automating re-labeling, and applying improvements in real-time to keep your RAG systems on an upward trajectory.

Integrate Feedback Loops for Continuous Improvement

Follow a repeatable process to continually evaluate and improve your RAG application

Jason Liu

Cohort 1

Cohort 2

Welcome to the most technically rigorous and hands-on Large Language Model (LLM) application course available today. This isn't just another AI course – it's your gateway to mastering the art and science of deploying production-grade LLM solutions that stand out in the real world. As part of Maven's Top-Rated Content, this course is designed for those who have already mastered the basics of RAG, cosine similarity, vector databases, and LLMs. We'll take you to the next level, focusing on practical aspects of packaging and deploying these models in real-world production environments. For cohort members joining this intensive learning experience, here's what you get: - 6 weeks of in-depth content- Weekly office hours for personalized guidance- Real-world projects and challenging assignments- Guest Lectures by Leading AI Professionals- Continued support post-graduation- Lifetime access to course materials What You'll Master: Course Highlights Agents: Forget CrewAI or Autogen, build your won Agents from scratch. Learn what it takes to make an Agent from the ground. Contribute to Open Source community. Advanced RAG Solutions: Dive into enterprise-level RAG architectures and learn how to build and implement semantic caching from scratch using GCP and Redis. LLM Hosting and Deployment: Gain insights into best practices for hosting Large Language Models (LLMs) in diverse production settings, creating inference endpoints, and deploying LLMs on serverless platforms. Continual Pre-Training and Fine-Tuning: Explore advanced techniques for continual pre-training, fine-tuning LLMs, and mitigating catastrophic forgetting. Learn how to build a data pipeline for pre-training, apply causal language modeling, and leverage scaling laws. Model Merging and Mixture of Experts: Master techniques for merging multiple models to enhance their collective capabilities, including the Mixture of Experts (MoE) approach. Learn to use tools like mergekit for efficient model merging. Quantization Methods: Discover techniques to reduce model size while maintaining performance, crucial for deployment in resource-constrained environments. Inference Speed Optimization: Learn strategies to accelerate inference speeds for real-time language processing, ensuring efficient and responsive AI systems. Responsible AI Implementation: Explore ethical AI development using guardrails like NeMo, Colang, and Llama Guard to ensure AI systems align with responsible AI principles. Agentic RAG and Chunking Strategies: Implement advanced semantic chunking techniques and explore AI agent frameworks like AutoGen to enhance the capabilities of RAG systems. DSPy and Knowledge Graphs: Learn to create and utilize knowledge graphs effectively, mastering DSPy as an alternative prompting approach for structured data handling and enhanced AI interaction. Throughout the course, we will analyze state-of-the-art AI products, reverse-engineering some through Python. As a bonus, you'll have access to experimental products being developed at Traversaal.ai, my startup, allowing you to stay at the forefront of advancements in the field. Prerequisite: You should have coding experience of building RAG Solution. Understanding of Encoders and Decoders and some knowledge of Cloud Solutions and APIs. If you feel the need for a more foundational course, consider checking out my other offering on LLMs: Building LLM Applications (https://maven.com/boring-bot/ml-system-design).

Go Beyond Basic Frameworks: Build and Deploy Production-Grade AI Solutions

Sundays

Work in teams to build solutions, this requires engagement with other team members

Weekly projects

Course schedule

Machine Learning Engineer exploring different techniques to scale LLM solutions

Researcher, who would like to delve in to various aspects of open-source LLMs

Software Engineer, looking to learn how to integrate AI into their products

This course is for you if you are a:

What you’ll get out of this course

Understand and implement complex AI architectures, including enterprise-level RAG systems and agentic RAG strategies. You will also dive deep into the Mixture of Experts (MoE) technique and other model merging strategies to enhance the capabilities of your AI systems.

Advanced AI Architectures

From building semantic caches using GCP and Redis to deploying LLMs on serverless platforms like AWS Bedrock, you'll learn the practical skills to deploy and manage AI applications in real-world scenarios.&nbsp;

Practical Skills for Deployment

Acquire advanced techniques for fine-tuning LLMs, enabling you to adapt these models to specific tasks or domains and enhance their performance in targeted applications.

Fine-Tuning Expertise

Explore strategies for exploring and optimizing inference speeds, ensuring that your language models perform efficiently in real-time scenarios, a crucial skill for deploying responsive and scalable applications.

Efficient Inference Processing

Understand the importance of ethical AI development and learn to implement guardrails using tools like NeMo, Colang, and Llama Guard to ensure your AI systems align with responsible AI principles.

Knowledge of Responsible AI

Build and Optimize Production-Grade RAG and LLM Applications: Master Advanced Techniques for Scalable, Secure, and Low-Latency AI Solutions

Cohort 4

Hamza Farooq

Cohort 3

Cohort 5

In this transformative course, you'll journey from being an AI novice to becoming a confident and skilled Generative AI Engineer. You'll gain a deep understanding of the fundamental concepts behind Large Language Models (LLMs) and their applications in solving real-world engineering challenges. Through hands-on projects and expert guidance, you'll master the art of prompt engineering, enabling you to effectively communicate with and guide LLMs to achieve desired outcomes. You'll explore cutting-edge techniques like Retrieval Augmented Generation (RAG) and fine-tuning, empowering you to build sophisticated AI systems that can process and generate human-like text. As you progress through the course, you'll develop the skills to create agentic systems that can autonomously perform complex tasks and make intelligent decisions. You'll also learn best practices for deploying and monitoring LLM applications in production environments, ensuring scalability and reliability. By the end of this course, you'll have a robust portfolio of Generative AI projects showcasing your ability to leverage LLMs for various engineering applications. You'll be equipped with the knowledge and practical experience needed to drive innovation in your organization and make a significant impact in the field of AI engineering. Some of the tools we will use during the course are: OpenAI chatGPT, Amazon SageMaker, Amazon Bedrock, Anthropic's Claude, LangChain, LlamaIndex, HuggingFace Whether you're a software engineer looking to expand your skill set, a data scientist seeking to harness the power of Generative AI, or a domain expert aiming to incorporate AI into your projects, this course will provide you with the tools and confidence to succeed. Join us on this transformative journey and unlock the limitless potential of Generative AI in engineering. Schedule at-a-glance: ⏰ Time:&nbsp;10:30 am - 12:00pm EST (New York time) P.S. Education should be accessible to everyone. If money is a barrier for you, please email (me@RiteshAI.com), and I will help by providing a discount. If you are from a developing country (e.g. India, Brazil), I will work with you to consider an appropriate discount. *disclaimer: credits are subject to change and you may not get full credit what vendor offers.

From AI Novice to Generative AI Engineer

Saturday & Sunday

Before each weekend, there will be a pre-work for students to be ready to understand the concepts - either in form of pre-recorded video or pre-reading of articles.

Pre-work

Use 1 hour weekly to prepare your project which you can demonstrate to your peers, bosses and the world.

Capstone project

Software Engineers and Developers:Expand your skills, build intelligent apps, automate tasks, and stay competitive in AI-driven development

Data Scientists and AI Enthusiasts:Explore Generative AI, leverage LLMs for insights, push boundaries, and solve real-world problems

Domain Experts and Researchers:Incorporate AI into your projects, automate tasks, gain insights, and collaborate with AI experts.

Who is this course for

<ul><li>Learn to effectively communicate with and guide LLMs through optimized prompts.</li><li>Unlock the full potential of LLMs and achieve desired outcomes in your AI projects.</li></ul>

Mastering Prompt Engineering

<ul><li>Discover how RAG enhances LLMs by integrating external knowledge sources.</li><li>Build powerful question-answering systems and knowledge-intensive applications.</li></ul>

Implementing Retrieval Augmented Generation (RAG)

<ul><li>Adapt pre-trained LLMs to your specific domain or task requirements.</li><li>Improve the performance and accuracy of your AI models through fine-tuning techniques.</li></ul>

Fine-Tuning LLMs for Domain-Specific Tasks

<ul><li>Create autonomous AI agents that can perform complex tasks and make intelligent decisions.</li><li>Explore the concepts of goal-oriented behavior and rational decision-making in AI systems.</li></ul>

Developing Agentic AI Systems

<ul><li>Learn best practices for deploying LLM applications in production environments.</li></ul>

Deploying LLM Applications

Master Generative AI for engineers: prompt engineering, RAG, fine-tuning & more. Build cutting-edge AI apps & revolutionize your projects.

Ritesh Vajariya

Vanguard

Explorers

As a data professional, you have learned to use tools, crunch numbers, develop attractive charts or dashboards, and create engaging presentations. But how do you know if your conclusions are right? What if, after all this effort, your conclusions are weak or even plain wrong? Data visualization can make data persuasive for your audience and for yourself—very persuasive! So much so that you may not realize that your thinking does not hold water upon closer scrutiny. Unfortunately, learning software tools, programming languages, design strategies, or presentation skills do not shield you from making reasoning mistakes. You may have heard of or learned to recognize some statistical fallacies, but you may lack a coherent and cohesive framework to think about reasoning fallacies in data visualization systematically. This is what I aim to provide you with Rhetorical Data Visualization (RhetVis). RhetVis aims to make you a real data detective who can identify reasoning gaps and provide solutions when assessing and designing new data visualizations for yourself and others. Using the RhetVis framework, you can identify and solve issues with 1) how data is collected/generated, 2) how data values are computed, transformed, and visualized, and 3) how data visualizations are framed using titles and other contextual elements. WHAT YOU CAN EXPECT RhetVis includes five modules:1. Introduction: Become familiar with the RhetVis framework.2. Data-reality gaps: Spot gaps in how the data has been generated.3. Data transformation: Spot issues with data selection, filtering, aggregation, etc.4. Visual representation: Spot problems with how the data is represented.5. Contextual factors: Become mindful of the framing effect of titles and other elements. Each week, we cover two modules for a total of three weeks of work. Each module includes:- 45-60 min recorded video lectures: Learn the basics of each module.- Lecture slides: Quickly review the course content.- Quizzes: Test your knowledge, spot comprehension gaps, and obtain clarifications.- 1.5 hours of live meetings: Discuss the main ideas and develop practical skills through group activities (using breakout rooms) and presentations.- Printable cheat sheets: Quickly review and recall the main elements of the RhetVis framework.- Reading material and resources: to dive deeper into the topics covered. The course also includes a final mini-project to develop individually and present to the group on the final day of the course. The course does not require any specific data processing or visualization tool. We will use Google Docs/Slides and drawing tools (paper, digital drawing apps, etc.). WHO WILL BENEFIT MOST FROM THIS COURSE - Data analysts and scientists who want to get stronger at reasoning with data visualization- Visualization designers who know how to build engaging/beautiful/effective visualizations but lack the skills to evaluate the validity of their data arguments- Data enthusiasts who want to become more skilled at critically evaluating the visualizations they encounter in the world WHO THIS IS NOT FOR If your main goal is to learn specific data visualization tools (Tableau, Power BI, etc.), specific programming languages, or how to create pretty/engaging visuals, this is not the course for you. The course is also inadequate for people who want to work at their own pace and are not interested in interacting with others. The live meetings require everyone to be active and engage in discussions, conversations, and collaborative work.

Avoid being fooled or fooling others with data

If your events are recurring and at the same time, it might be easiest to use a single line item to communicate your course schedule to students

Tuesdays & Thursdays

Feel free to type out dates as your title as a way to communicate information about specific live sessions or other events.

May 7, 2022

Schedule items can also be used to convey commitments outside of specific time slots (like weekly projects or daily office hours).

Data analysts and data scientists who want to avoid reasoning fallacies with data and data visualizations.

Visualization designers who want to avoid misleading or confusing their readers.

Data enthusiasts who want to become better data thinkers and avoid being misled by data visualizations.

When you look at a visualization, how do you know if you can trust its message? Thinking critically means learning to assess the validity of a data visualization message and spot limitations and fallacies. We will review, discuss, and assess many data visualizations together.

Think critically about visualizations and their messages.

We often take data at face value, but problems with data often start when the data is collected. You'll learn a systematic way to spot gaps between data and the reality it represents. We will conduct specific hands-on activities to spot the gaps learned in the lecture.

Spot and correct gaps between data and the reality it represents.

Before data is visualized, it always goes through some selections and transformations. How do these decisions affect the validity of a visualization? What kind of fallacies can these choices generate? We will conduct hands-on activities to learn how to spot the fallacies.

Detect and avoid interpretation fallacies in data selection and transformation.

Have you ever noticed that the same data can be visualized in many different ways? How do you decide which way works best? You'll learn to reason about the effect of design choices. We will design alternative representations of exactly the same data to analyze their effect.

Dissect visualizations into their individual parts and identify the elements that impact their interpretation.

There is so much focus on the graphical part of data visualization, but how about the contextual elements, such as titles and annotations? You'll learn to think about bias and framing. We will design alternative titles and annotations to evaluate their effect.

Assess bias and framing effects from titles and annotations.

Spot and avoid data fallacies, recognize biases, and create more truthful visualizations through use cases, hands-on activities, and more.

Lightning Lessons

Build a Document Processing Pipeline for RAG Systems