Anny He

Software consultant, speaker, AI educator

A gentle introduction to open weight models

So you can pick the best model for your use case

How to evaluate open-weight models against your use case

Go from overwhelm to shortlisted candidates

Which models to use for multi-modal vs. text-only 

We'll explore local providers (eg. ollama) and remote ones (eg. AWS)

Trade-offs of running models locally vs. hosted remotely

Eliminate uncertainty in shipping AI features

GenAI Product Builder @ Google, ex-Meta

Marily Nika, Ph.D AI/ML

Senior Data, ML and GenAI Scientist

George Zoto

Senior NLP/AI Scientist

August-2026

“Does it work?”... “Is it good enough?”... “Can we ship it?”...How do you answer these questions for AI products? You’re responsible for “running evals” but what does that mean?How do you choose the right metrics, interpret fuzzy results, and make a confident decision?This course gives you a framework to do just that.<ul><li>Map user value to evaluation (eval) objectives so your metrics aren’t abstract. Define success then translate it into measurable criteria.</li><li>Choose metrics you can actually maintain: capability, safety, UX friction, latency, cost and “does this reduce support tickets or increase activation.”</li><li>Set ship/no-ship thresholds you can defend to leadership.</li><li>Build lightweight workflows that work in real teams: human review where it matters, automation where it lasts, documentation that drives decisions.</li><li>Consider domain constraints (e.g., healthcare safety) and know what to avoid: silent failures, misleading proxy metrics and tests that don’t reflect production.</li><li>Tie everything to ROI: impact vs unit cost, eval coverage vs reliability, and the minimum viable monitoring you need post-launch.</li></ul>Experience AI evals through a case-based approach with a real AI product that we evaluate together.

Acquire and develop a critical skill for product managers who are leading and contributing to AI products.

Make confident ship or hold decisions for AI features

Translate user value into clear evaluation goals

Choose the right metrics for capability, safety, UX, and cost

Set defensible thresholds leadership will trust

Build lightweight evaluation workflows teams can maintain

Navigate domain constraints and avoid common failures

PMs leading AI features, growth, or platform initiatives

PMs who partner with ML teams and want to set evaluation standards

PMs who need to make clear “ship or hold” calls without doing the engineering

Build, fine-tune, and evaluate an LLM in one live workshop.

Instructor @ Linux Foundation/DSR

Deloitte

FlixBus

The Linux Foundation

Data Science Retreat

ODSC

Daniel Voigt Godoy

Amazon best-selling author, Instructor @ Linux Foundation/Data Science Retreat

Mini-Batch #1

Most LLM fine-tuning advice is either too theoretical or assumes large teams, large datasets, and expensive infrastructure.This workshop shows a smaller, practical path using open-weight models: choose one narrow task, prepare a focused dataset, train a QLoRA adapter on accessible hardware, and evaluate whether it performs well enough to reduce cost, latency, or reliance on proprietary APIs.By the end of the workshop, you will have:✅ A clean task-specific fine-tuning dataset.✅ A trained LoRA/QLoRA adapter.✅ A before-and-after evaluation against the base model.✅ A reusable notebook you can adapt to your own data.✅ A decision checklist for when fine-tuning is worth doing versus when prompting, RAG, or better data is enough.This workshop is taught in a small cohort, so you will have the time and the opportunity to discuss your own use case, get feedback on whether it is a good fit for fine-tuning, and ask questions about your data, constraints, and next steps.

Become the person who knows when fine-tuning is worth it, how to run the first controlled experiment, and how to evaluate the results.

Choose a fine-tuning task worth testing

Build a task-specific fine-tuning dataset

Train a LoRA/QLoRA adapter end to end

Evaluate whether fine-tuning actually helped

Diagnose what to change after the first run

Reuse the workflow for future fine-tuning experiments

Python coding

Jupyter Notebook

You should understand prompts, inputs, outputs, and common model limitations.

Basic LLM prompting

Technical founders prototyping AI products

Applied AI / ML Engineers

Backend engineers building LLM features

Go from a demo and a feeling to an evaluation practice you can run every week and defend to whoever controls the budget.

I build AI that has to survive an audit

Amazon Web Services

HashiCorp

VMware

NGINX

University of California

Thomas Underhill

Product & Eng Leader (AWS, VMware, HashiCorp, NGINX) · Adjunct Professor

Cohort 1

Your team put a model in production and it seems to be working. Then someone above you asked what the AI spend is returning, and the honest answer was a demo and a feeling.That is not a failure of rigor on your part. It is a structural problem with a name. On complex work, checking an answer costs about as much as producing it, so grading does not scale, so nobody grades. Ninety six percent of engineering teams use AI tools. Around twenty percent measure the result specifically.The gap is not laziness. It is that the standard advice does not work on the problems you have. Accuracy misleads on imbalanced data. Your eval set is probably contaminated and you cannot check. And for the most valuable work your system does there is no right answer to compare against, which is where most teams give up and let a model grade a model.There are ways out of all of that, and none is new. Metamorphic testing dates to 1998 and was never pointed at language models until recently.This course builds one evaluation practice, for one real system of yours, across five weeks. You leave able to say what working means, prove whether you have it, and defend the number to whoever holds the budget.

You shipped it. Now somebody wants a number, and you do not have one.

Define what working means for output that changes every run

Build an evaluation set you can actually trust

Choose a metric you can defend, and set the threshold from cost

Grade the work that has no right answer to compare against

Catch a regression before a customer does

Defend the whole thing to the person holding the budget

Every exercise runs against a real system of yours. Without one there is nothing to evaluate, and the coursework does not work in theory.

An AI system already in production, or about to be

The exercises run on my data, but the plan is for your system. You need to know what it does, what failure costs, and who cares about it.

A system you are accountable for, and enough context to describe it

Every exercise is a design decision rather than an implementation. You will read code. Your engineers build what you specify.

No machine learning background, and no code required

Engineering leader who shipped it and cannot measure it.

Product leader being asked what the AI spend returns

Anyone whose team grades model output with another model

Build an AI First Operating Model for ROI and Impact

Create your AI First Blueprint

MASSIVUE

Microsoft

Barclays

Standard Chartered

Aimia Inc

Sandeep Joshi

7× Microsoft MVP · 26 years turning AI ambition into P&L impact across domains

A 4-hour working session for transformation and strategy leaders whose AI pilots have stalled. You'll leave having filled in the AI Operating Model Canvas for your own organization - nine blocks, one page, from ambition to measurable execution.

Cohort Next

95% of enterprises get zero return on their AI investment. Not because the models don't work, because AI gets funded, prioritized, and governed like a traditional IT project, bolted onto an operating model built for human throughput. The pilots impress. Then they never reach the P&amp;L.This session works the operating model itself, using the Protum AI Operating Model Canvas: three layers, nine building blocks, from AI Ambition at the top to a Measurement System at the bottom. You'll fill it in for your own organization, score yourself against the six capabilities of an AI-first company, and find the one block where your ROI is leaking. You leave with a completed canvas, an Investment Thesis one-pager for a real initiative, and a 90-day move with an owner and a metric.You do the work in the room. Not a framework you read about - your canvas, filled in, with the next move already decided.

Where AI actually creates competitive advantage for your org (Block 1) — and which decisions to Protect, Reshape, or Replace (Block 2)

Fill in the full AI Operating Model Canvas for your own organization

Diagnose exactly where your AI ROI is leaking

Score your organization against the six capabilities of an AI-first company

Write an Investment Thesis that survives your CFO

Design a Governance Model for autonomous and agentic work

Come with one company you can map - your own or your clients. The canvas gets filled with real context.

A real organization in mind

At least one AI effort underway - piloted, planned, or plateaued - gives you something concrete to diagnose. Not mandatory though.

A stalled or scaling AI initiative helps

You won't write code or configure tools. If you can read a business case, you're ready.

No technical skills needed

Chief Transformation Officer, VP/Director of Transformation, Head of Digital Strategy, Transformation Program Director

Head of Enterprise Architecture, Principal Architect, Consultants, Coaches and IT Leaders

COO, VP Operations, Head of Business Change, senior People/Org-Design leader with P&amp;L or org-design responsibility

Agentic AI

Coding with AI

AI Workflows

Claude Code

OpenClaw

Vibe Coding

AI Evals

AI Transformation

RAG & Search

AI for PMs

AI for Engineers

AI for Designers

AI for Marketers

AI for Founders

Product Sense

Product Discovery

User Research

Prototyping

Growth

Analytics

Tech Foundations

Strategy

Influence

Leadership

Career Growth

Machine Learning

LLM Ops

Context Eng

Security

System Design

Figma

Design Systems

Visual Design

Design Strategy

Automation

Content Marketing

Demand Gen

Go-to-Market

Product Marketing

Positioning

Social Media

Brand

B2B Marketing

SEO & AEO

AI for Leaders

AI Governance

Communication

Management

People Operations

Exec Presence

Storytelling

Goal-setting

Personal Brand

Fundraising

Everyone

Operators

Data Scientists

Business Analysts

User Researchers

Customer Success

Project Managers

HR Professionals

Sales People

Lawyers

Finance

Investors

Real Estate

Educators

Creators

Engineers and engineering leaders who build, deploy, and scale software systems.

Data scientists who apply statistical and ML techniques to extract insights from data.

Applying AI tools and techniques to professional workflows — assign alongside more specific AI topics.

Operating and deploying large language models in production — fine-tuning, inference, monitoring, and cost optimization.

A gentle introduction to open weight models

What you'll learn

Why this topic matters

You'll learn from

Anny He