Marcin Grzejszczak

Observability of your Application

Broadcom Inc.

VMware

4Finance

Java Champion, maintainer of Spring Cloud, Micrometer projects

Learn how to identify performance bottlenecks and common runtime issues in modern distributed systems.

Diagnose Latency and Errors in Real-World Applications

Understand how to instrument your application using Micrometer to gain visibility into performance and behavior.

Use Micrometer to Correlate Metrics, Traces, and Logs

Learn how to interpret logs, traces, and metrics together to efficiently locate and resolve production issues.

Use Observability Data to Pinpoint Root Causes

Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas. Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products and Features (MVPs) and make the organization evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment. You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law. The 2nd week covers additional topics, some more technical, including cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluations, AI/Machine learning and triggering, scaling, build vs. buy, challenges, and requested topics. The advanced topics course is a follow-on to this: https://bit.ly/AdvancedABFS

Empower your organization to be data-driven and innovative

Course syllabus

Three x 2-hour sessions + 1/2-hour bonus in week 1 on Monday, Tuesday, and Thursday.The bonus time is optional. Material may be more technical, and time for additional Q&amp;A

Week 1: Mon, Tue, Thu

Two x 2-hour sessions + 1/2-hour bonus in week 2 on Monday and Thursday

Week 2: Mon, Thu

Optional Q&A

Course schedule

Data science managers and scientists will be able to design and interpret the experiment results in a trustworthy manner

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents through A/B tests

Who is this class for?

What you will learn from this 10-hour course

You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!

Understand and internalize the humbling reality that we are poor at assessing the values of ideas: most ideas fail!

Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.

Understand the key advantages and limitations of A/B testing

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.

Learn how to design metrics and the Overall Evaluation Criterion

Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.

Learn how to designing trustworthy A/B tests

Learn about the cultural challenges, the humbling results (most ideas fail, pivoting, iterating, learning), institutional memory, ideation, prioritization, experimentation platforms

Learn about the cultural challenges

When building AI or machine learning models, using A/B testing and triggering to evaluate the models that were built offline based on historical data

Learn about the relationship to AI and Machine Learning Modeling

When you can't run an A/B test, quasi-experimentation methods and the risks of observational causal studies

Learn about complementary quasi-experimental techniques

What are key challenges and open questions in the field

Learn about existing challenges

If there is something specific you want to cover, there is time allocated for topics voted by the audience to discuss

Learn about YOUR topic of interest

The course focuses on developing the intuition and common misunderstandings, without the details of the statistics, which you can find in many books. We cover p-values, statistical power, and triggering.You can go as technical as you want in the Q&amp;A and community discussions

Technical to your needs

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business

Microsoft

Airbnb

Amazon

Dr. Ronny Kohavi

Executive at Microsoft, Airbnb, Amazon, author of best-selling A/B testing book

7 July 2025

8 Sept 2025

Transform your retrieval from "good enough" to "mission-critical" in weeks, not months Most RAG implementations get stuck in prototype purgatory. They work well for simple cases but fail on complex queries—leading to frustrated users, lost trust, and wasted engineering time. The difference between a prototype and a production-ready system isn't just better technology, it's a fundamentally different mindset. The RAG Implementation RealityWhat you're experiencing right now: ❌ Your RAG demo impressed stakeholders, but real users encounter hallucinations when they need accuracy most❌ Engineers spend countless hours tweaking prompts with minimal improvement❌ Colleagues report finding information manually that your system failed to retrieve❌ You keep making changes but have no way to measure if they're actually helping❌ Every improvement feels like guesswork instead of systematic progress❌ You're unsure which 10% of possible enhancements will deliver 90% of the value What your RAG system could be:With the RAG Flywheel methodology, you'll build a system that: ✅ Retrieves the right information even for complex, ambiguous queries✅ Continuously improves with each user interaction✅ Provides clear metrics to demonstrate ROI to stakeholders✅ Allows your team to make data-driven decisions about improvements✅ Adapts to different content types with specialized capabilities✅ Creates value that compounds over time instead of degrading What Makes This Course DifferentUnlike courses that focus solely on technical implementation, this program gives you the systematic, data-driven approach used by companies to transform prototypes into production systems that deliver real business value: ✅ The Improvement Flywheel: Build synthetic evaluation data that identifies exactly what's failing in your system—even before you have users✅ Fine-tuning Framework: Create custom embedding models with minimal data (as few as 6,000 examples)✅ Feedback Acceleration: Design interfaces that collect 5x more high-quality feedback without annoying users✅ Segmentation System: Analyze user queries to identify which segments need specialized retrievers for 20-40% accuracy gains✅ Multimodal Architecture: Implement specialized indices for different content types (documents, images, tables)✅ Query Routing: Create a unified system that intelligently selects the right retriever for each query The Complete RAG Implementation Framework Week 1: Evaluation SystemsBuild synthetic datasets that pinpoint RAG failures instead of relying on subjective assessments BEFORE: "We need to make the AI better, but we don't know where to start."AFTER: "We know exactly which query types are failing and by how much." Week 2: Fine-tune EmbeddingsCustomize models for 20-40% accuracy gains with minimal examples BEFORE: "Generic embeddings don't understand our domain terminology."AFTER: "Our embedding models understand exactly what 'similar' means in our business context." Week 3: Feedback SystemsDesign interfaces that collect 5x more feedback without annoying users BEFORE: "Users get frustrated waiting for responses and rarely tell us what's wrong."AFTER: "Every interaction provides signals that strengthen our system." Week 4: Query SegmentationIdentify high-impact improvements and prioritize engineering resources BEFORE: "We don't know which features would deliver the most value."AFTER: "We have a clear roadmap based on actual usage patterns and economic impact." Week 5: Specialized SearchBuild specialized indices for different content types that improve retrieval BEFORE: "Our system struggles with anything beyond basic text documents."AFTER: "We can retrieve information from tables, images, and complex documents with high precision." Week 6: Query RoutingImplement intelligent routing that selects optimal retrievers automatically BEFORE: "Different content requires different interfaces, creating a fragmented experience."AFTER: "Users have a seamless experience while the system intelligently routes to specialized components." Real-world Impact From Implementation﻿✅ 85% blueprint image recall: Construction company using visual LLM captioning✅ 90% research report retrieval: Through better text preprocessing techniques✅ $50M revenue increase: Retail company enhancing product search with embedding fine-tuning✅ +14% accuracy boost: Fine-tuning cross-encoders with minimal examples✅ +20% response accuracy: Using re-ranking techniques✅ -30% irrelevant documents: Through improved query segmentation Join 400+ engineers who've transformed their RAG systems with this methodology Your Instructor Jason Liu has built AI systems across diverse domains—from computer vision at the University of Waterloo to content policy at Facebook to recommendation systems at Stitch Fix that boosted revenue by $50 million. His background in managing large-scale data curation, designing multimodal retrieval models, and processing hundreds of millions of recommendations weekly has directly informed his consulting work with companies implementing RAG systems.

Stop building RAG systems that impress in demos but disappoint in production

A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.

A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user

Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.

This Course Is For You If You Are

By the end of this course, participants will be able to

Implement the Data and Evals Flywheel approach to continuously develop and improve RAG applications—breaking free from guesswork and relying on measurable, iterative enhancements.

Adopt a Systematic, Data-First Methodology

Quickly assess your retrieval systems using precision and recall metrics, identify bottlenecks, and confidently validate changes without sinking into endless trial-and-error cycles.

Run Fast, Unit-Test-Like Evaluations

Generate and utilize synthetic data sets to speed up experimentation, enabling you to test new approaches, embeddings, and architectures before committing full resources.

Leverage Synthetic Data for Rapid Iteration

Apply fine-tuning strategies for embedding models to boost search relevance and explore hard negative examples to further sharpen retrieval performance.

Master Fine-Tuning & Hard Negative Mining

Use query classification and segmentation techniques to pinpoint exactly where your RAG system falls short—whether it’s due to limited inventory or insufficient capabilities.

Classify Queries & Identify Bottlenecks

Create tailored indices for documents, images, tables, SQL databases, and more. Learn when and how to fuse or layer these indices to handle complex retrieval tasks elegantly.

Design Specialized Indices for Multiple Modalities

Implement synthetic text chunk generation and strategic summarization methods to improve retrieval results, ensuring end-users get clear, concise, and contextually rich answers.

Enhance Retrieval with Summarization & Chunking

Develop systems that intelligently route queries to the right indices, tools, or pipelines. Blend and fuse indices effectively to handle nuanced, multi-step queries.

Implement Query Routing & Index Fusion

Evaluate the performance of your routing logic and each individual index separately. Gain the nuance to fine-tune global system performance and local retrieval accuracy in tandem.

Optimize Both Global & Local Performance

Design explicit and implicit feedback mechanisms—capturing user signals, automating re-labeling, and applying improvements in real-time to keep your RAG systems on an upward trajectory.

Integrate Feedback Loops for Continuous Improvement

Follow a repeatable process to continually evaluate and improve your RAG application

Jason Liu

Cohort 3

Cohort 4

NOTE 1: The price shown on this page is in _USD_.NOTE 2: Check out the incentive program on this page for ways to save money.NOTE 3: LangGraph, Cursor, and n8n are the default tools we use, but you are not constraint to those You can book me to talk about this bootcamp or anything else AI / career / startup related:https://calendly.com/amirfzpr/20min-ama 🟢 WHAT WILL I BE ABLE TO BUILD IN THIS BOOTCAMP?You’ll build a functional application powered by LLM agents to automate a workflow of your choice. Along the way, you'll learn to design agent workflows, and integrate required tools. By the end of the program, you'll have a working application ready to showcase in a public demo. You can see this playlist for examples of what previous students built: https://tinyurl.com/llm-agent-demos 🟢 HOW WILL YOU HELP ME BUILD APPS LIKE THESE IN JUST 7 WEEKS?This program is all about learning by doing. You do a real project, applying concepts you learn about AI agents with our expert guidance. Specifically:👨🏽‍🏫 1:1 sessions with the us to help you think through bigger picture questions👨‍🎓 Async and sync sessions with a dedicated mentor for constant support, code reviews, and real-time feedback We also provide the "agents playbook" a step by step guide for building agentic systems with lots of hints, tips, and resources. 🟢 OK! THAT DOES SOUND FUN, BUT HOW DO I KNOW IF I HAVE THE PRE-REQUISITES?We will provide tutorials and curated material that can help you build regardless of how you want to approach it (coding, vibe coding, or no-code tools). So what you really need to be successful in this bootcamp is a lot of desire to build and not much else. You could also leverage our freely available playbook to prep beforehand and come ready (scroll down on this page). 🟢 HMM! SO I DON'T NEED TO BE A CODER?If you are, then your freedom to build what you want will definitely increase. However, we have seen people do successful projects during the bootcamp using no-code tools like N8N, LangFlow, Dify etc. We have also seen folks "vibe coding" using Windsurf, Cursor, Co-pilot etc.There are options! The question is, are you ready to build? 🟢 I SEE! SO, WHAT SHOULD I EXPECT TO HAPPEN DURING?Think about it like this (since we're developing a product here, eh?): This is a 3-sprint product dev + 1 week of prep for the public demo.🖌️ SPRINT 1 - DESIGN (weeks 1 &amp; 2)🔨 SPRINT 2 - DEVELOP (weeks 3 &amp; 4)🚢 SPRINT 3 - DEPLOY (weeks 5 &amp; 6)🎭 PUBLIC DEMO (week 7)In each sprint, you communicate your progress or blockers on Slack, you collaborate on your designs / code on GitHub, you show up to provide weekly project updates and get feedback, and finally you do an end of sprint demo. 🟢 OK! HOW WOULD I KNOW IF I'M MOVING IN THE RIGHT DIRECTION?You will get feedback async via Slack and other platforms we use, and also you join weekly group office hours, and when you need you can book the mentors for a 20 min 1:1 call.In addition, each sprint will wrap up with a "sprint demo" where you can show what you've done up to that point, and your peers and the mentors will provide feedback. 🟢 WHAT IF I DON'T HAVE A PROJECT IDEA?We provide a default project and 3 series of tutorials that provide various ways that project can be implemented using:🤃 coding,🤄 vibe coding, and🤅 no-code tools. Each series is led by one of our mentors and will meet live for an hour every week to build together and you can follow along. At the end of each session, the mentor would point out ways that you can add to the project and make it your own. If you follow through and engage it's likely that you will end up with an interesting spin on the default project that you can show off in the public demo. 🟢 COOL! I'M EXCITED TO GET STARTED, AND A DISCOUNT WOULD REALLY SWEETEN THE DEAL!Glad to hear that! There are a few options:🧑🏿‍🤝‍🧑🏼 You can find a buddy or colleague and sign up together. (20% off 2-9 seats&nbsp;|&nbsp;25% off 10+ seats)✴️ We often share discount codes with communities (including AISC members - reach out to community@ai.science if you're not part of our slack). So, check in with the communities you are part of.📆 We regularly hold events and info sessions where we share flash sales discounts valid for a day or two. Join the mailing list on this page to receive notifications about those.👨🏽‍🏫 You can apply to our Incentive Program that provides up to 75% refunds (not discounts). Apply here: https://forms.gle/YnvsoREuEX35DAxv8

Mentorship Program to Help You Build an Agent-based MVP

Controlling the behavior of your agents often requires coding to include various tools, memory, or orchestration.

Python Coding

If you don't want to code, you can use no-code agent studios. We will provide specific resources but general familiarity is useful.

No-code App Building

If you are planning to code (or vibe code), familiarity with one of the coding assistants (GitHub copilot, cursor, etc), is helpful.

Coding co-pilots

You want to build an agentic MVP to test the feasibility of your startup idea.

You want to build an agentic MVP to explore the new feature you want to add to your existing product

You want to build an agentic MVP to demonstrate your skills for career progression purposes.

This bootcamp is for you if...

At the end of this bootcamp you will ...

Grasp how multiple LLMs can collaborate with each other, get feedback from or assign tasks to humans, ad use tools to tackle complex problems

Understand and utilize the core ingredients of multi-agent LLM systems

So, you build a system but does it work? How can it be improved? Out of 100 things you can change, which one is more important to focus on?

Understand the role of evaluation in building a robust agentic system

Start from a workflow, design an agentic system for it, build it, evaluate it, find test cases to fix, repeat. You do this multiple times in the 7 weeks and those reps are how you build the mental muscle!

Leave with a worked out muscle of experimentally developing an agentic system

Design, develop, deploy, and demo an LLM agent application in 7 weeks!

Cohort-2507

Amir Feizpour, PhD

10 yrs in NLP, Founder of AISC - Community of 5k AI devs and founders

Cohort-2505

2510

🚨 THIS IS IS OUR FINAL LIVE COHORT - (We have to get back to building AI applications!) - Enrolled students get LIFETIME ACCESS to all materials and recordings!🚨 Do you catch yourself asking any of the following questions while building AI applications? 1. How do I test applications when the outputs are stochastic and require subjective judgements? 2. If I change the prompt, how do I know I'm not breaking something else? 3. Where should I focus my engineering efforts? Do I need to test everything? 4. What if I have no data or customers, where do I start? 5. What metrics should I track? What tools should I use? Which models are best? 6. Can I automate testing and evaluation? If so, how do I trust it? If you aren't sure about the answers to these questions, this course is for you. This is a hands-on course for engineers and technical PMs. Ideal for those who are comfortable coding or vibe coding. ---WHAT TO EXPECT This course will provide you with hands-on experience. Get ready to sweat through exercises, code and data! We will meet two times a week for four weeks, with generous office hours (read below for course schedule). We will also hold office hours and host Discord community where you can communicate with us and each other. In return, you will be rewarded with skills that will set you apart from the competition by a wide margin. (see testimonials below). All sessions will be recorded and available to students asynchronously.--- COURSE CONTENT Lesson 1: Fundamentals &amp; Lifecycle LLM Application Evaluation- Why evaluation matters for LLM applications - business impact and risk mitigation- Challenges unique to evaluating LLM outputs - common failure modes and context-dependence- The lifecycle approach from development to production- Basic instrumentation and observability for tracking system behavior- Introduction to error analysis and methods for categorizing failures Lesson 2: Systematic Error Analysis- Bootstrap data through effective synthetic data generation- Annotation strategies and quantitative analysis of qualitative data- Translating error findings into actionable improvements- Avoiding common pitfalls in the analysis process- Practical exercise: Building and iterating on an error tracking system Lesson 3: Implementing Effective Evaluations- Defining metrics using code-based and LLM-judge approaches- Techniques for evaluating individual outputs and overall system performance- Organizing datasets with proper structure for inputs and reference data- Practical exercise: Building an automated evaluation pipeline Lesson 4: Collaborative Evaluation Practices- Designing efficient team-based evaluation workflows- Statistical methods for measuring inter-annotator agreement- Techniques for building consensus on evaluation criteria- Practical exercise: Collaborative alignment in breakout groups Lesson 5: Architecture-Specific Evaluation Strategies- Evaluating RAG systems for retrieval relevance and factual accuracy- Testing multi-step pipelines to identify error propagation- Assessing appropriate tool use and multi-turn conversation quality- Multi-modal evaluation for text, image, and audio interactions- Practical exercise: Creating targeted test suites for different architectures Lesson 6: Production Monitoring &amp; Continuous Evaluation- Implementing traces, spans, and session tracking for observability- Setting up automated evaluation gates in CI/CD pipelines- Methods for consistent comparison across experiments- Implementing safety and quality control guardrails- Practical exercise: Designing an effective monitoring dashboard Lesson 7: Efficient Continuous Human Review Systems- Strategic sampling approaches for maximizing review impact- Optimizing interface design for reviewer productivity- Practical exercise: Implementing a continuous feedback collection system Lesson 8: Cost Optimization- Quantifying value versus expenditure in LLM applications- Intelligent model routing based on query complexity- Practical exercise: Optimizing a real-world application for cost efficiency GUEST SPEAKERS ---1. Teresa Torres (Product Talk) : From Noob to 5 Automated Evals in 4 Weeks (as a PM).2. Jeremy Howard &amp; Johno Whitaker (AnswerAI): SolveIt -&nbsp;The Thinking Developer's Environment (a great environment for evals!)3. Leonard Tang (Haize Labs): Scaling Inference-Time Compute for Better LLM Judges4. Charles Frye (Modal): Taming diffusion QR codes with evals and inference-time scaling5. Udi Menkes (Product Manager, Intuit): A playbook for building AI agents you can trust.6. Skylar Payne:&nbsp;10x Your RAG Evaluation by Avoiding these Pitfalls TOOLS TRACK: Eval vendors will provide optional tutorials---- Harrison Chase: LangSmith- Mikyo King &amp; Isaac Flath: Arize Phoenix- Wayde Gilliam: Braintrust

Eliminate the guesswork of building AI applications with data-driven approaches.

If your events are recurring and at the same time, it might be easiest to use a single line item to communicate your course schedule to students

Monday & Wednesday & Friday

Feel free to type out dates as your title as a way to communicate information about specific live sessions or other events.

May 7, 2022

Schedule items can also be used to convey commitments outside of specific time slots (like weekly projects or daily office hours).

Weekly projects

Engineers &amp; Technical PMs building AI products who are interested in moving beyond proof-of-concepts.

Those interested in moving beyond vibe-checks to data driven measurements you can trust, even when outputs are stochastic or subjective.

Founders and leaders who are unsure of the failure modes of their AI applications and where to allocate resources.

Who this course is for

What you’ll get out of this course

We've tried all of them so you don't have to.

Acquire the best tools for finding, diagnosing, and prioritizing AI errors.

And how to best leverage data when you do have users.

Learn how to bootstrap with synthetic data for testing before you have users

Data flywheels ensure you have examples to draw from for prompts, tests, and fine-tuning.

Create a data flywheel for your applications that guarantees your AI will improve over time.

How can you really trust LLM-as-a-judge? How should you design them? We will show you how. We will also show you how to refine prompts, generate metadata, and other tasks with the assistance of AI.

Automate parts of your AI evaluation with approaches that allow you to actually trust and rely on them.

We will show you approaches to discover all the ways AI is not performing

Ensure your AI is aligned to your preferences, tastes and judgement.

There are an infinite number of things you can try, tests you can write, and data you can look at. We will show you a data-driven process that helps you prioritize the most important problems so you can avoid wasting time and money.

Avoid common mistakes we've seen across 35+ AI implementations.

We will provide end-to-end exercises, examples and code to make sure you come away with the skills you need. We will NOT just throw a bunch of slides at you!

Hands-On Exercises, Examples and Code

Generous office hours ensures students can ask questions about their specific issues and interests.

Personalized Instruction

Learn proven approaches for quickly improving AI applications.  Build AI that works better than the competition, regardless of the use-case.

Cohort 2

Hamel Husain

ML Engineer with 20 years of experience.

Shreya Shankar

ML Systems Researcher Making AI Evaluation Work in Practice

Cohort 1 has $1,000 in modal credits.  This is NOT guaranteed for Cohort 2

Lightning Lessons

Observability of your Application