Haroon Choudery

Master Evaluation Techniques for LLM Apps

OpenAI

 Intercom

Retool

Airtable

CEO, Autoblocks AI

Learn what role evaluations play in LLM apps and why they are crucial ensuring their effectiveness & reliability.

Why evaluations are necessary

Explore how to select the best evaluation techniques (i.e., rules-based evals or LLM judges) for your LLM use case

How to choose the right evaluations

Discover methods, like fine-tuning & expert alignment, for improving the consistency and accuracy of your evaluations.

Improving evaluation reliability

"Ishan takes a black box, rips it apart to understandable compartments, and clearly communicates what is going on. Highly recommend." - Joe, CFO ______________________________________________________________________ Why This Course Stands Out: You already know how vital AI is to your career—it's in every conversation, project, and strategy session. But diving into AI feels daunting with limited time for self-study or complex programming. That’s where this course comes in. Skip the endless prep work and gain an intuitive, hands-on understanding of AI and transformers in just a few hours—without needing an advanced degree. No other course offers such an interactive, fast, and in-depth technical understanding of AI using an actual GPT-2 model, all using an easy-to-learn spreadsheet interface. ______________________________________________________________________ Perfect for Busy STEM Professionals:Whether you're a developer, technical manager, or executive, this course demystifies AI and teaches you exactly how Large Language Models (LLMs) like GPT-2 work, with no prerequisites beyond a STEM background. It’s ideal for: • Software/Web Developers seeking an easy introduction to machine learning. • Science &amp; Tech Professionals navigating AI transformations in their fields. • Technical Sales/Marketing crafting messaging for AI products. • Product/Project Managers overseeing LLM-powered solutions. • Executives/Leaders guiding AI strategy in their organizations. ______________________________________________________________________ What You’ll Learn:Using a unique spreadsheet implementation of GPT-2 (the predecessor to ChatGPT), we’ll break down each model component with crystal-clear explanations. In just a few sessions, you’ll be able to: • Understand how LLMs work—as quickly as possible. • Explain every key component—tokenization, embeddings, attention, and more. • Gain AI literacy—so you can confidently lead and manage AI-driven teams. • Build a foundation for further AI projects—and even more advanced learning. ______________________________________________________________________ Why Spreadsheets?You don’t need to be a coder or a calculus expert to understand AI—just the right approach. This course uses an innovative, no-code, spreadsheet-based method to teach LLMs. It's fast, interactive, and designed for busy professionals like you. By breaking down complex AI principles into easy-to-follow spreadsheet exercises, you’ll gain a lasting, foundational understanding of how large language models work. This is knowledge you can carry across into any model, any programming language, and any project. ______________________________________________________________________ Hands-On Learning, No Programming Required: By the end of the course, you’ll have a working, detailed understanding of how transformers power LLMs like ChatGPT, Claude, and Llama. Plus, you'll have a spreadsheet tool you can explore and tweak to deepen your understanding further. ______________________________________________________________________ See it for Yourself: Still unsure? Watch the FREE lightning lesson:"Inside the Brain of an LLM: How it Works in 30 Minutes" to see how accessible AI learning can be: https://maven.com/p/eeea6f/inside-the-brain-of-an-llm-how-it-works-in-30-minutes ______________________________________________________________________ Join the Community of Learners: • “I could’ve read 20 papers on this and not gotten nearly the understanding I got from this hands-on course.”— Class Participant from Adobe • “Highly recommend this course for anyone wanting to build a foundation in LLMs. Ishan’s interactive approach makes complex concepts incredibly clear.”— Matt, Business Intelligence Analyst ______________________________________________________________________ Practical Details: • Course Access: All sessions are recorded and available to stream after the course concludes, so you’ll never miss a thing.• Reimbursement: Your company may cover this course if you're building or transitioning into AI at work. ______________________________________________________________________ New for this cohort! We've dropped the requirement for Excel. We have a browser based alternative. Some exercises may use Google Sheets and/or Python.

Skip the mountains of prep work and learn how AI works in just a few hours

Provide a brief description for each module as a way to sell the value of what students will be learning throughout the course.

First Module

Second Module

Third Module

Course syllabus

If your events are recurring and at the same time, it might be easiest to use a single line item to communicate your course schedule to students

Tuesdays & Thursdays

Feel free to type out dates as your title as a way to communicate information about specific live sessions or other events.

May 7, 2022

Schedule items can also be used to convey commitments outside of specific time slots (like weekly projects or daily office hours).

Weekly projects

Course schedule

Executives, engineering managers, and product managers leading and coordinating technical AI teams leveraging LLMs

Developers without formal training in machine learning that want to quickly understand the LLM technology in their projects

A gentle introduction to LLMs for professionals pivoting their career to AI from traditional software engineering or other STEM fields

Anyone who can read a spreadsheet that is motivated and deeply interested in how the next wave of computing works

Who is this course for

What you’ll get out of this course

This course breaks down complex AI concepts into easy-to-follow steps using Excel, making it accessible for anyone with a STEM background.

Understand how LLMs work as quickly as possible

Precisely understand in detail the architecture and function of the GPT2 model, including tokenization, embeddings, attention, multi-layer perceptron, and layernorm.

Recognize and explain every component of an LLM

You'll gain a solid understanding of how LLMs function, enabling you to confidently engage with engineers and technical stakeholders.

AI Literacy for managing technical teams

Build a foundation that equips you with the essential knowledge to explore advanced AI topics, undertake AI projects, or pursue further coursework.

Prepare yourself for further exploration, projects, or coursework in artificial intelligence

Master AI Concepts with a Real LLM Implemented Entirely in a Spreadsheet. No Complex Math. Just Clarity. 

Ishan Anand

December 4-13, 2024

Jan 8—17, 2025

🟢 WHAT WILL I BE ABLE TO BUILD IN THIS COURSE?You’ll build a functional application powered by LLM agents to automate a workflow of your choice. Along the way, you'll learn to design agent workflows, and integrate required tools. By the end of the course, you'll have a working MVP ready to showcase in a public demo. You can see this playlist for examples of what previous students built: https://tinyurl.com/llm-agent-demos 🟢 HOW WILL YOU HELP ME BUILD APPS LIKE THESE IN JUST 7 WEEKS?This course is all about learning by doing. You do a real project, applying concepts you learn about AI agents with our expert guidance. Specifically:👨🏽‍🏫 1:1 sessions with the instructors to help you think through bigger picture questions👨‍🎓 Async and sync sessions with a dedicated TA for constant support, code reviews, and real-time feedback To help you prepare, we have also curated 70+ hours of content, a "Learning Path" covering everything from product design and coding fundamentals to advanced LLM engineering concepts that you can consume at your own pace, as soon as you sign up. (see the next question for details) 🟢 OK! THAT DOES SOUND FUN, BUT HOW DO I KNOW IF I HAVE THE RRE-REQUISITES AND I'M READY FOR BUILDING?That's the most common question we get! The "Learning Path" we have curated should also give you a good sense of what knowledge we assume as pre-reqs. You should get started on that and then sign up for the course as soon as you feel you are ready:https://maven.com/p/bafd83/llm-agents-learning-path 🟢 I SEE! SO, I'LL DO MY PREP BEFOREHAND, THEN WHAT SHOULD I EXPECT TO HAPPEN DURING?Think about it like this (since we're developing a product here, eh?): This is a 3-sprint product dev + 1 week of prep for the public demo.🖌️ SPRINT 1 - DESIGN (weeks 1 &amp; 2)🔨 SPRINT 2 - DEVELOP (weeks 3 &amp; 4)🚢 SPRINT 3 - DEPLOY (weeks 5 &amp; 6)🎭 PUBLIC DEMO (week 7)In each sprint, you communicate your progress or blockers on Slack, you collaborate on your designs / code on GitHub, you show up to provide weekly project updates and get feedback, and finally you do an end of sprint demo. 🟢 OK! HOW WOULD I KNOW IF I'M MOVING IN THE RIGHT DIRECTION?You will get feedback async via Slack and other platforms we use, and also you join weekly group office hours, and when you need you can book the teaching staff for a 20 min 1:1 call. In addition, each sprint will wrap up with a "sprint demo" where you can show what you've done up to that point, and your peers and the teaching staff will provide feedback. We also will invite guest experts for each of the demos to provide additional feedback:🖌️ SPRINT 1 - Sr. AI Product Experts🔨 SPRINT 2 - Sr. ML Engineers🚢 SPRINT 3 - Sr. MLOps Engineers🎭 PUBLIC DEMO - Investors, Experienced Founders, Corporate Directors 🟢 WHAT IF I DON'T HAVE A PROJECT IDEA?We will have a default project (most probably related to automating data science work) that you can participate in. If your goal is to better internalize the concepts rather than working on a specific idea, you can join that project. We commonly see that people re-take the course in the future to further develop specific ideas they have, and focus on building the foundations the first time they take the course. 🟢 COOL! I'M EXCITED TO GET STARTED, AND A DISCOUNT WOULD REALLY SWEETEN THE DEAL!Glad to hear that! There are a few options:🧑🏿‍🤝‍🧑🏼 You can find a buddy or colleague and sign up together. (20% off 2-9 seats&nbsp;|&nbsp;25% off 10+ seats)✴️ We often share discount codes with communities (including AISC members - reach out to community@ai.science if you're not part of our slack). So, check in with the communities you are part of.📆 We regularly hold events and info sessions where we share flash sales discounts valid for a day or two. Join the mailing list on this page to receive notifications about those.

Mentorship Program to Help You Build an Agent-based MVP

We will have group discussions and guests lecturers on Thursdays, and project office hours on Sundays.

Thursdays & Sundays

Course will run for 7 weeks in January and February. You will have access to reading / watching material, instructor, and hands-on teaching assistants.

Jan. 13, 2025 - Mar. 1, 2025

There will be a project that you will do alongside the course material with weekly milestones.

You want to build an agentic MVP to demonstrate your skills for career progression purposes.

You want to build an agentic MVP to test the feasibility of your startup idea.

You want to build an agentic MVP to explore the new feature you want to add to your existing product

This course is for you if...

At the end of this course you will ...

Grasp how multiple LLMs can collaborate with each other, get feedback from or assign tasks to humans, ad use tools to tackle complex problems

Understand the core principles of multi-agent LLM systems

Bridge the gap by crafting strategies to seamlessly integrate LLMs with existing products, boosting user experience, productivity, and business metrics.

Integrate LLMs with existing product offerings to enhance user and customer experience

Learn what it takes to build robust multi-agent LLMs. Understand security best practices and ensure system reliability.

Understand key considerations for securing, robust, and reliable LLM systems

Design, develop, deploy, and demo an LLM agent application in 7 weeks!

Amir Feizpour, PhD

10 yrs in NLP, Founder of AISC - Community of 5k AI devs and founders

Abhimanyu Anand

Cohort 2409

Cohort-2501

You're going to create an "Insights Extraction AI App" ("Unstructured Text Data" to "Structured Insights") using a modern stack of tools and AI engineering patterns An end-to-end deployed (live on the web) application that converts unstructured data (podcast, meeting notes, customer feedback, essays, etc.) and outputs structured insights (tweets, product descriptions, summaries, etc.). The goal is to build &amp; learn a flexible application structure so you can extend to your use case. Example applications you'll be able to extend this course's code to: Meeting "Next Steps" extractor, Podcast Summarizer, Customer Survey Trends Extractor, Extract class lessons from lectures, Data stream classifier, Custom apps with your own data AI Engineering Patterns we'll cover:* Prompting - The basic 3 step pattern for writing development prompts + optimizers, structured outputs* Observability - To manage costs, tracing and debugging* Retrieval/Chains - Adding more context to your LLMs and connecting multiple LLM calls together for a unified outcome* Model Choice - Tribal knowledge about when to pick certain models over another (spoiler: it depends on your use case)* Evals - The new style of unit tests for your LLM apps* Mindset - You'll learn how to approach novel problems with AI We promise three outcomes with this course:* You’ll launch an AI-powered app&nbsp;designed to teach you the foundations to build your own tools* You’ll surround yourself with a new&nbsp;group of friends:&nbsp;other builders who are as passionate as you (and hold you accountable)* You’ll 2x your likelihood to ship by experiencing first-hand the greatest trait a builder can have:&nbsp;a bias for action You'll get experience building with a modern AI powered development stack:* Cursor - Write code, but with ChatGPT by your side* v0 - Build user interfaces with words, not code* LLMs - Outsource intelligence to Large Language Models APIs like OpenAI/Anthropic* LangChain/LangSmith - Orchestrate and observe your LLM calls, ensuring your costs don't run out of control* Railway/Vercel - Deploy your apps* Supabase - Simple postgres database* Zapier - Simple ways to quickly integrate your application with other tools* Anthropic Artifacts - For prototyping mini applications "The best way to get good at something is usually to just practice actually doing the thing in question. A lot of very capable people outsmart themselves with complex plans that involve working a lot on fake prerequisites." - Sam Altman 10/27/2024 It’s no longer skill level that's holding you back, it’s just your desire to take a step forward. Along the journey, you’ll have access to mentors, successful builders that have built apps, a network of like-minded people, creative workshops, and 1:1 sessions. This program is a fast-paced, highly engaging program, but you don’t need to take time off work to participate. We’ve designed the program to be as flexible as possible for those who work full-time or have busy schedules. With that said, we encourage you to commit to&nbsp;at least&nbsp;4-5 hours of your spare time each week, for deep work, feedback groups, and live sessions. In addition to the course material, this cohort includes:* Code Templates - All materials shown in class will be available for download so you can continue to quickly build after the class* Slack - A channel for you to ask questions, connect with other builders, and get support on your projects* Guests - Each week we'll feature an expert from the industry to talk about AI Engineering This course is not:* Learn to be a programmer from scratch* Learn the absolute basics* Learn how to fine tune LLMs On the traditional SWE side, we will build our backend with Python and Postgres, frontend with Next.js, and deployed on Railway. Though this course is taught in python, you have the flexibility to bring whatever tool stack you're comfortable with. Who you are* You have a intermediate level understanding of Python + Javascript (React is a plus)* You're not afraid to roll up your sleeves and learn new technology* You value shipping something quick* You want to build AI-powered products* You're ready to break a big problem down into shippable chunks

Explore AI engineering patterns while building real systems

Saturdays

We'll kick off the course in the new year

Jan 6, 2024

Each week will consistent of live instruction, video overviews, and "homework" (the fun kind) through practicing to build.

Scrappy Developers. You're a hands-on builder ready to turn your skills into real AI applications, even if development isn't your day job

Technical Leaders. CTOs and technical managers who are struggling to elevate the AI conversations with their teams

Experienced Developers. Software Engineers who want to build AI products but haven't dove into an AI focused tool stack yet

Become proficient in prompting, observability, chaining, model choice, structured outputs and scaling apps

Playbook of best practices for 5 areas of LLM development

You'll build your own tool which will start saving you time from Day 1. We'll walk through the steps of deploying your application on a custom domain (Ex: YourCoolAppYouBuilt.com)

A deployed "Insight Extraction" app live on the web

You'll gain hands-on experience with Cursor, v0, LLMs (OpenAI, Anthropic, Google), LangChain &amp; LangSmith, Railway/Vercel, Supabase, Zapier, and Anthropic's Artifacts

A survey of modern AI tools that will speed boost your workflow

We will run through step by step instructions on how to reduce the cost and latency of your LLM apps while maintaining (or even increasing) performance. This is critical to keep your LLM costs under control

Instructions to scale your LLM apps

Learn the key AI engineering patterns by building an app that extracts insights from unstructured text. Practical AI insights for developers

Greg Kamradt

Founder @ Leverage, AI Educator

Cohort 1

If you’re a software engineer or data scientist, chances are you’ve built or seen a proof of concept (POC) for a generative AI app. It’s exciting at first—an impressive demo that showcases the potential of large language models. But here’s the problem: that’s where it usually stops. These POCs often fail to scale into reliable, production-ready applications. The result? Endless iteration cycles, unreliable outputs, and frustration as teams struggle in what we call “POC purgatory.” It doesn’t have to be this way. With the right tools, principles, and processes, you can move beyond POCs and build LLM-powered applications that work—reliably, robustly, and at scale. In this course, we’ll show you how. You’ll not only learn the technical skills to design, test, and deploy generative AI applications but also develop the product mindset needed to iterate and improve rapidly. This isn’t just about building something cool—it’s about creating systems that deliver real value, solve real problems, and scale with confidence. What to ExpectThis is a highly interactive course designed to get you results. Across live workshops, discussions, and hands-on projects, you’ll build your skills while solving real-world challenges. Expect: •	Live coding sessions to design and refine LLM applications.•	Iteration exercises to explore how small changes can dramatically improve reliability.•	Discussions on user-centric development and integrating feedback loops.•	Expert-led Q&amp;As to help you tackle your toughest challenges. What You’ll Learn •	How to transform POC demos into production-grade applications.•	Strategies for managing non-determinism and optimizing outputs through prompt engineering.•	How to monitor, log, and debug AI systems to ensure reliability and performance.•	Best practices for handling structured outputs and integrating function calling into your applications.•	Building workflows for iterative development and experimentation.•	Practical skills through hands-on app development, including creating a Generative AI app to query PDFs. Bonus Perks All students will receive $1,000 worth of Modal credits to help power their AI applications, with more API credit announcements coming soon. Who This Course Is For This course is designed for: •	Data scientists and machine learning engineers who are sick of unreliable POCs and want to ship reliable LLM applications.•	Software engineers who want to learn how to build generative AI systems and master the LLM software development lifecycle. Who This Course Is Not For This course is not for those who are: •	Looking for an introductory course in AI—some programming knowledge is required.•	Expecting ready-made solutions for AI problems without active participation.•	Unable to commit to hands-on projects, live discussions, and iterative development exercises. Why Take This Course By the end of this course, you’ll: •	Understand the first principles of generative AI development.•	Gain hands-on experience building robust, scalable applications.•	Learn how to troubleshoot, monitor, and optimize AI systems for production.•	Develop workflows to iterate and refine AI applications effectively.•	Leave with the confidence to deploy reliable AI systems at scale.

Ground principles that underpin successful GenAI app development

Live in person sessions. We'll finalize session times by mid-November.

Mondays and Wednesdays

To ensure hands-on practical time, there will be project work to complete throughout the course.

Data scientists, machine learning engineers who are sick &amp; tired of seeing and building prototypes &amp; want to ship reliable LLM applications

Software engineers who want to learn how to build Generative AI systems and learn the LLM software development lifecycle.

You’ve seen impressive generative AI demos—but how do you take them to production? In this course, you’ll learn the exact workflows, tools, and engineering practices needed to turn concepts into reliable, scalable applications that deliver real value.

Move beyond POCs and build production-ready AI systems

Struggling with inconsistent outputs? Through interactive exercises, you’ll explore proven techniques for optimizing prompts and ensuring your AI delivers actionable results in production settings.

Master prompt engineering for reliable results

When your AI application breaks, you’ll know exactly how to fix it. We’ll walk you through real-world debugging sessions, monitoring techniques, and tools to ensure your systems stay reliable at scale.

 Learn to debug and monitor AI systems like a pro

Want your AI to produce precise, usable results? You’ll learn how to handle structured outputs and function calls, making your applications smarter and more user-friendly.

Build smarter applications with structured outputs

Stop wasting time in endless iteration cycles. Through guided projects, you’ll discover how to set up workflows that refine your applications faster and with less frustration.

Accelerate your development process with iterative workflows

Tired of theoretical knowledge? You’ll build a fully functional app that uses text and image models to query PDFs, giving you practical experience solving complex real-world problems.

Develop hands-on expertise by building GenAI and LLM apps

Whether you’re looking to lead AI projects, impress stakeholders, or level up your technical expertise, this course gives you the tools to stand out as an AI/ML practitioner.

Advance your career with practical, proven techniques

These aren’t just skills for demos—they’re skills for delivering scalable, production-ready systems that solve real-world business challenges.

Solve Real Problems at Scale

Leave the course with a project you can add to your portfolio, demonstrate to your boss, or use to showcase your ability to deploy reliable AI systems.

Showcase Tangible Results

Build LLM-powered software reliably & from first principles. Learn the GenAI software development lifecycle: agents, evals, iteration & more

Lightning Lessons

Master Evaluation Techniques for LLM Apps