Claude

ChatGPT

LangChain

CrewAI

Microsoft

Sol Farahmand

AI Engineer | 2X Entrepreneur | AI Hackathon Winner

How to Trace and Monitor AI Agents with LangSmith

MindStudio

Lovable

AI Hackathon Winner | 2X Entrepreneur | AI Workflow Builder

Learn to see why checking only an agent's final output hides where a failure actually happened.

Why Final Answers Don't Show What Went Wrong

Learn to add LangSmith tracing to a LangChain agent running locally through Ollama, no code changes needed.

Set Up LangSmith Tracing On A Local Agent

Learn to inspect a LangSmith trace to see which tools ran, how long each step took, and token usage.

Read A Trace To See Tool Calls And Latency

Build robust LLM-powered apps, chatbots, and agents. Learn by writing real code, one line at a time.

CEO @ Actualize

Jay Wengrow

Software Engineer and Educator, Author of A Common-Sense Guide to AI Engineering

Cohort 2

Cohort 3

Do you have a grasp of basic LLM principles but struggle to put it all into code? Tired of scattered documentation, outdated tutorials, and frameworks that impress in demos but collapse in production?You’re not alone. Building with large language models doesn’t have to be confusing - it can be clear, structured, and deeply rewarding.In this course, you’ll build real LLM-powered apps, one line of code at a time. You'll see how an app is built from scratch in real time, and follow along. Along the way, you'll discover not only how to implement these systems, but the why behind each line of code.From chatbots to agents and RAG to evals, you’ll learn the core concepts of building atop LLMs. More importantly, you’ll get how LLMs “think” - allowing you to guide them to do what you want despite their nondeterministic nature. Additionally, you’ll be able to balance quality, latency, and cost, making big-picture decisions about AI-powered app architecture.AI engineering isn’t just another branch of software development - it’s a different mindset altogether.

Gain hands-on AI Engineering proficiency by writing real code

Gain real-world intuition for working with LLMs

Reliably guide LLMs instead of getting unpredictable results.

Build retrieval-augmented assistants that can access proprietary knowledge

Use evals to systematically optimize your LLM app’s performance over time

Assemble agents that act upon the real world

This course is designed for current software engineers who will use Python  to build LLM-powered apps. We won't teach basic coding here.

Software Engineering

We'll be using Python, but we'll keep it simple in case you're more familiar with other coding languages.

Note about Python

If you want an inside look at what's involved with building AI apps but don't plan on coding yourself, you'll still follow what's going on.

Exception: Product Managers

Software engineers who want to build LLM-powered chatbots and agents, or break into AI engineering more generally

Data Scientists/Engineers who want to build their own LLM-powered apps

Product Managers who want to AI engineering literacy to dream bigger and spec sharper

You stop arguing about whether the agent is good, and start showing the number, the rubric behind it, and the decision it justifies.

Author, AI Agent Evaluation (Springer)

Abdullah Mansoor

Co-author of AI Agent Evaluation (Springer 2026), written for non-coders.

Cohort 1

You shipped an AI agent. It demos beautifully. Then a board member, a customer, or your own gut asks the question you have been avoiding: how do you know it works? Most teams answer with more testing, more prompt tuning, and a dashboard nobody trusts. The gap does not close, because the missing piece is not effort. It is a system: a written standard for what good means, an instrument that measures against it, and a loop that runs on a schedule. This is the companion course to AI Agent Evaluation (Mansoor and Bokhari, Springer 2026), cut for the person who makes the release decision rather than writes the code. Six modules take you from naming the failure you are afraid of, to writing a contract your agent has to honor, to scoring it on four dimensions, to running a loop that improves it every week. Every tool, framework and metric is introduced at concept level. After a module you can hold your own with your engineers using these terms. No lesson asks you to open a terminal. You leave with seven artifacts built on your own agent: a contract, a rubric with anchors, a benchmark set, a failure taxonomy, a release gate, a cost and latency budget, and a seven-step playbook.

A demo proves it worked once. Production asks you to prove it works every time.

Name the failure pattern your agent is most likely to hit

Write a one-page agent contract

Score your agent on the four dimensions that decide readiness

Build a rubric two independent raters would score the same way

Run the Biruni Loop so quality improves on a schedule

Put cost and latency on the same row as quality

Every exercise is built on your own agent, so you leave with a contract and a rubric for something real rather than a worked example.

An AI agent in production, or one you are about to ship

Everything is taught at concept level. You will hold your own with your engineers using these terms without opening a terminal.

No coding required, and no prior testing or evaluation background

The templates only pay off against a decision you actually have to make, so bring a release you are genuinely unsure about.

Willingness to bring one real agent and one real release decision

Founders whose agent demos beautifully, then answers the same question two ways on a Tuesday. You need the rate, not the anecdote.

Product leads who own the AI feature but not the code. Engineers report a score. You cannot tell if the product improved or the test eased.

Technical founders who stopped coding. You read a system diagram fine. You want the vocabulary back at concept level, without a code tour.

Everyone is learning to build AI agents. Few can prove one is safe to deploy.

Founder | Agentic AI Governance Advisor

Cyber Pros Training

Centene

ISACA

Trenton Health Team

François B. Arthanas

Founder of CyberProsAI | Agentic AI Governance Advisor | CISSP, CISA, AAIA™

Learn how to govern agentic AI systems before they create risk, fail audits, or get your org in trouble.

AI agents are not chatbots. A chatbot answers a question. An AI agent retrieves customer records, calls tools, triggers workflows, issues refunds, sends emails, and acts sometimes before a human ever reviews the output.That means the question landing on YOUR desk is different:Can this agent leak PII? → Can it be prompt-injected through a poisoned document? → Can it call the wrong tool or the right tool with the wrong authority? → Can it issue an unauthorized refund? → Who owns the risk? And what evidence proves the controls actually work?Most professionals freeze at this question because they've only ever seen AI governance as policy slides and framework acronyms. They've never actually run an agent risk review end to end.In this one-day sprint, you will. You'll take a realistic enterprise AI agent through the complete practitioner loop scope → risk → control → test → evidence → decision and finish the day having written and defended a real deployment recommendation.You will watch an AI agent get broken live. Then you'll learn exactly how to make sure it never happens on your watch.

In one day, you'll run a real AI agent risk review start to finish and leave able to do it again at work, with the templates to prove it.

Scope a real enterprise AI agen a support agent that can access customer data

Break it: Run a live red-team session against the agent (no coding required)

Turn your findings into a one-page Trust Snapshot leadership can act on

Make the call and defend it in a live mock review board

You do not need to be an engineer. You do not need to know how to code. You do not need advanced AI expertise.

There are no strict prerequisites for this course

If you are curious about AI agents and want a practical way to think about oversight, this workshop will meet you where you are and move you

What you do need is an interest in understanding how AI is changing governance

This workshop is for professionals who know AI agents are becoming important and do not want to get left behind.

It is built for GRC, IT, cyber, privacy, legal &amp; consultants who need to understand how agentic AI changes the governance conversation

You should join this workshop if you want to understand how AI agents create risk &amp; how to contribute meaningful value inside your org

Build and reason about enterprise agents grounded in your team's knowledge base

Agentic Search Expert

Shopify.com

Reddit

Wikipedia

LexisNexis

Apple

Doug Turnbull

Led teams at Shopify, Reddit, Wikipedia

AI Builder/Educator (6 Million Students)

Google

Instagram

OpenAI

Netflix

Yale

Hugo Bowne-Anderson

AI & data engineer, consultant, educator of 6+ million students (ex-Yale)

August 2026

Your company sits on endless reports, documents, and files. Agents leverage your company’s rich volumes of data to support customers and make business decisions.In this course agentic search expert Doug Turnbull partners with AI expert Hugo Bowne-Anderson to create agents that actually work.At the foundation of production agents sit an LLM, a reasoning loop, an agent harness, and a layer of context engineering + retrieval.Research consistently shows AI answer quality depends more on the underlying search than any other factor. We’ll go from the ground up - building an agent from scratch, giving you what you need to make them actually useful with accurate search + retrieval.You will leave with a working e-commerce agent, a retrieval pipeline, a starter eval set, and a deployment-ready architecture you can adapt to your own company data.Enroll before July 7 to get very early bird pricing at $750. Early bird pricing increases to $850 on July 7 and the full course price of $950 begins on August 1.

Real agents don’t just chat: they find, organize, and act.


Build agents from first principles

Ground agents in enterprise knowledge

Engineer reliable agent harnesses

Evaluate, automate, and deploy agents

Data scientists, machine learning engineers who are sick &amp; tired of seeing and building prototypes &amp; want to ship reliable LLM applications

Software engineers who want to learn how to build Generative AI systems and learn the LLM software development lifecycle.

Search technologists who connect enterprise knowledge with AI systems with RAG + Agentic Serach

Agentic AI

Coding with AI

AI Workflows

Claude Code

OpenClaw

Vibe Coding

AI Evals

AI Transformation

RAG & Search

AI for PMs

AI for Engineers

AI for Designers

AI for Marketers

AI for Founders

Product Sense

Product Discovery

User Research

Prototyping

Growth

Analytics

Tech Foundations

Strategy

Influence

Leadership

Career Growth

Machine Learning

LLM Ops

Context Eng

Security

System Design

Figma

Design Systems

Visual Design

Design Strategy

Automation

Content Marketing

Demand Gen

Go-to-Market

Product Marketing

Positioning

Social Media

Brand

B2B Marketing

SEO & AEO

AI for Leaders

AI Governance

Communication

Management

People Operations

Exec Presence

Storytelling

Goal-setting

Personal Brand

Fundraising

Everyone

Operators

Data Scientists

Business Analysts

User Researchers

Customer Success

Project Managers

HR Professionals

Sales People

Lawyers

Finance

Investors

Real Estate

Educators

Creators

Introduction and Motivation Poll

Importance of Monitoring AI Agents

Monitoring Architecture with LangChain and LangSmith

Data Privacy and LangSmith Data Collection

Python Code Walkthrough for LangChain Agent

Environment Setup: Python, Ollama, and Virtual Environment

LangSmith Configuration and API Keys

Testing the Agent and Analyzing Traces in LangSmith

LangSmith Monitoring Dashboard Overview

Upcoming Sessions and Shared Resources

Q&A: Security, LangGraph, and Crew AI

Engineers and engineering leaders who build, deploy, and scale software systems.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Building autonomous AI agent systems that take actions, use tools, and orchestrate multi-step workflows.

Operating and deploying large language models in production — fine-tuning, inference, monitoring, and cost optimization.

How to Trace and Monitor AI Agents with LangSmith

What you'll learn

Why this topic matters

You'll learn from

Sol Farahmand