Improve Your Under-performing AI Agentic Product

Arpy Dragffy

Founder of PH1, Host of Product Impact

Brittany Hobbs

Research expert for AI-powered products

Solve the problems that evals and telemetry can't

This is a four-week sprint for the researchers, strategists and product leaders now accountable for whether a conversational AI product delivers value to customers. PH1 runs a complete study on one real product in front of the cohort, with every recording, transcript and finding open to you, while you run the same study on your own product. You leave with the evidence to say what isn’t working and what to change.

Conversational AI breaks the principles most product teams learned on. The same question returns different answers, users can ask anything, and many failures never produce a complaint. Dashboards and evals report on what they were built to see, so a product can look healthy while customers quietly give up on it.

This is not yet another Evals course, the automated test suites your engineers run against an AI product. If you want one of those we highly recommend taking any course by Hamel and Shreya.

Expect this to be part group-counselling, part focus group amongst friends. We’re here to share everything we’ve learned working with Spotify, Microsoft, National Football League, BBVA, and many more.

What you’ll learn

Go from guessing whether your AI product works to proving it, and leading the research and evals that make it better.

  • Learn why deterministic product principles break when the same question returns a different answer each time

  • Audit a real AI product using PH1's checklist, demonstrated live in week 1

  • Find where your product's moat sits, and how it erodes once the model is a commodity

  • Learn who to speak to, and why power users, loud users and representative users tell you different things

  • Draft your interview questions and get PH1's review for bias and focus before you interview

  • See why probabilistic products mislead research for different reasons, and need different fixes

  • Run AI-moderated interviews with your own users, with recruitment and interview costs included

  • Set your KPIs before you write the script, so the study answers what you need to know

  • Watch PH1 run the same study on a real product, from first interview to final findings

  • Find the gaps in telemetry and the silent failures that never produce a complaint

  • Set journey-based KPIs that hold up when your product answers differently each time

  • Use adversarial research to find where your product breaks

  • Learn when to use evals, and what separates great evals from poor ones

  • Turn your study findings into golden data sets and failure modes your engineers can use

  • Improve your evals through continuous testing and feedback loops

  • Build an improvement operating system in Claude Code or Codex, backed by a promptable repository of every cohort study

  • Adapt the method for regulated industries and for high-trust or high-risk product

  • Review your improvement plan with PH1 in an optional private session after week 4

Learn directly from Arpy & Brittany

Arpy Dragffy

Arpy Dragffy

20 years research & strategy for Spotify, Microsoft, NFL, health, education

Brittany Hobbs

Brittany Hobbs

Build the right product, get the telemetry to make better decisions

Google, BBVA, Hims and hers, Dell
PH1 Research
See all products from Arpy

Who this course is for

  • UX researchers and strategists adding conversational AI to their practice

  • Design directors and design leads responsible for AI experiences

  • Product managers who own a chatbot, a customer-facing agent, or an agent that routes work between people

What's included

Live sessions

Learn directly from Arpy Dragffy & Brittany Hobbs in a real-time, interactive format.

Lifetime access

Go back to course content and recordings whenever you need to.

Community of peers

Work alongside up to 11 researchers, strategists and product leaders facing the same questions about their AI products, in a closed room where everyone has signed an NDA.

Certificate of completion

Show your team and your network that you can test and improve a conversational AI product.

AI-powered research studies with your own users

Learn how to build, launch and analyse AI-powered research studies that give you critical insights at scale to improve your product. Recruitment and interview costs are included.

Hands-on audit practice

Audit a real AI product with PH1's checklist, and learn where conversational AI fails and what to look for.

Weekly office hours

Book time with an instructor every week to work through questions about your product, your study or your team.

Private review of your improvement plan

Close the course with an optional 30-minute session with PH1 to review your plan for improving your product and your evals.

Maven Guarantee

Your purchase is backed by the Maven Guarantee.

Course syllabus

12 lessons • 4 projects

Week 1

Feb 2—Feb 7

    When good products fail & why

    2 items

    Homework

    2 items

Week 2

Feb 8—Feb 14

    When research misleads & why

    2 items

    Homework

    2 items

Frequently asked questions

Maven for Teams

Reimbursement

Get your company to pay

Everything L&D needs: email template, receipts, and certificate of completion.

Get reimbursed

Team discount

Learn with your teammates

Save 20%+ when 2 or more teammates enroll in the same cohort.

Save 20%+ with a team

Private cohort

Run a cohort for your org

A dedicated cohort with a custom schedule and curriculum, tailored to your team.

Book a private cohort

$1,800

USD

Feb 2—Feb 27
Enroll