Hall Research AI

Cohort 1

Summer 2024

Validating and testing RAG systems to manage generative AI risks in real-world use.

ASRC Feb 2025

ASCR Spring 2026

Learn to validate and govern RAG-based generative AI with cutting-edge human-calibrated testing and NIST AI 600-1 guidance.

Patrick Hall

Principal Scientist at HallResearch.ai, GWU Faculty

Agus Sudjianto

Senior VP, H2o.ai and Former EVP & Head of Model Risk, Wells Fargo

Governing GenAI - From Prompts to Policy

Software Systems Architect

Legal Director

Model validator

Director, AI Division

Senior Data Scientist

Director - Model Risk

Lead Quantitative Analyst

In coordination with ASRC Federal, HallResearch.ai offer a second AI Ethics Technical Training Course.

Leader in Responsible AI & Risk Mgmt

Prof. Patrick Hall

Principal Scientist, HallResearch.ai and CAIO, GW School of Business

ASRC AI Ethics Technical Training Course

Program Manager

The New York Times

WIRED

FORTUNE

National Institute of Standards and Technology

Like other transformative technologies, artificial intelligence and machine learning (AI/ML) create both significant opportunity and meaningful risk. Recent advances in generative AI and emerging agentic systems have expanded what is technically possible—while also raising new governance, safety, and reliability challenges. To operate effectively in this rapidly evolving landscape, mission-driven practitioners need a grounded understanding of modern AI techniques—from supervised learning to large language models (LLMs)—along with the ability to evaluate real-world capabilities, manage risks, and implement systems responsibly. This training will align with the NIST AI Risk Management Framework (AI RMF), emerging federal guidance on generative AI, and established model risk management practices. Content is designed to be practical and system-agnostic, with applications across supervised, unsupervised, and generative AI systems. Special consideration is given to software engineers seeking deeper exposure to machine learning concepts and evaluation methodologies.

Stay at the forefront of AI/ML research, implementation, and risk management

Governance and Risk Management

Model Training, Selection, and Evaluation

Fairness and Transparency

Security and Privacy

Data Scientist/ML Engineer

In today's rapidly evolving artificial intelligence landscape, engineers and risk managers play pivotal roles in ensuring innovation and accountability. Our course on explainable AI (XAI) equips professionals with essential skills to navigate this rapidly expanding complex terrain. Students will gain hands-on experience exploring compliance considerations derived from real-world applications across diverse sectors. By merging technical expertise with compliance insights, this course empowers individuals to drive impactful, transparent AI solutions in their respective domains, fostering innovation and integrity. This course explores XAI, covering essential background definitions and concepts, explainable feature engineering, the diverse ecosystem of XAI models, post-hoc explanation methods, and the latest developments in audit and transparency laws and regulations. Some featured topics in this course on&nbsp;explainable AI (XAI): • The&nbsp;fANOVA&nbsp;framework• From&nbsp;penalized GLM to EBM and GAMinet•&nbsp;Monotonic GBMs•&nbsp;Surrogate model approaches•&nbsp;Explainable feature engineering•&nbsp;LOCO, LOFO, and pertubation• SHAP: The good, bad, and ugly• Security and bias considerations Background information and definitions are sourced from the National Institute for Standards and Technology (NIST), the National Academies, and other authoritative sources. The instructor teaches technical subjects with Python examples and compliance lessons based on his real-world experience in consumer finance, employment, and other regulated applications of ML. (Note: The instructor is not an attorney.) This course is designed primarily for data scientists and ML engineers. Technical risk executives or risk managers may find this course helpful in providing an updated overview of newer ML approaches suited for high-stakes applications. This course may interest others too. Materials may help regulators or policy professionals gain insights into the current state of ML technologies that can be used to comply with laws, regulations, or standards. If you're coming to ML from physics, econometrics, or psychometrics, this course can help you learn how to blend newer ML techniques with established domain expertise and notions of validity or causality.

Learn to build accurate, transparent, and interpretable ML/AI models. 

Course syllabus

Live lectures followed by discussion session.

Monday, July 8, 2024

Live lectures and demos followed by discussion session.

Wednesday, July 10, 2024

Monday, July 15, 2024

Wednesday, July 17, 2024

Course Schedule

ML engineers and data scientists who want to learn about XAI.

Technical risk executives seeking an updated overview of newer ML approaches suited for high-risk applications.

Policy professionals interested in the current state of ML technologies that may be used to comply with regulations and standards.

Who is this course for

Learning Objectives

<ul><li>Ecosystem of cutting-edge explainable ML models</li><li>Essential compliance awareness for ML transparency</li><li>Emergent audit and transparency laws and regulations</li></ul>

Build accurate, transparent, and understandable ML models.

<ul><li>Best practices for post-hoc explanation</li><li>Explainable feature engineering</li><li>Testing explanation quality</li></ul>

Interpret and explain sophisticated ML algorithms, enabling them to uncover insights, identify biases, and understand complex decisions.

Be equipped to build safer and more trustworthy AI systems, tackle real-world problems confidently, and drive positive impact in various domains such as healthcare, finance, and human resources.

Build safer and more trustworthy AI systems.

Learn to build accurate, transparent, understandable ML models. Get real-world policy and compliance insights for high-risk applications.

Principal Scientist, HallResearch.ai                                               
 & Assistant Professor, George Washington University

This expert-led online course offers a rigorous, practice-based approach to generative AI validation and governance. Participants will learn how to evaluate RAG systems using information retrieval techniques, embedding-based diagnostics, and human-in-the-loop testing—without defaulting to external benchmarks. The course also covers red-teaming and field testing strategies for assessing real-world risk scenarios. Grounded in NIST AI 600-1 guidance and the instructors' experiences in high-stakes domains like consumer finance, this course is designed for professionals who want substance over buzzwords.

Validating Generative AI: Practical Risk Management for RAG Systems and Beyond

Our techniques are not quick hacks. They're for practitioners who want to put the work in to build robust systems for high-impact domains.

Desire to build robust, high-impact AI systems

Our techniques aren't simple. You'll need some knowledge of language models, machine learning, and Python to get the most out of the course.

Experience with AI, machine learning, and Python

We don't build slop. This class is for people who understand the best tech is achieved through testing and review.

Exposure to technology governance approaches

<ul><li>Model Governance and Generative AI (Patrick Hall)</li><li>Formal Validation of Retrieval Augment Generation (RAG) Systems (Agus Sudjianto)</li></ul>

Module 1: July 22, 2025

<ul><li>Formal Validation of Retrieval Augment Generation (RAG) Systems (Continued) (Agus Sudjianto)</li></ul>

Module 2: July 23, 2025

<ul><li>Red-teaming for Real-world Assessment of Text-based Generative AI Systems (Patrick Hall)</li><li>Field Testing for Real-world Assessment of Text-based Generative AI Systems (Patrick Hall)</li></ul>

Module 3: July 24, 2025

Course schedule

Quantitative and Financial Analysts who need to understand how generative AI impacts model-driven investment and risk strategies.

Data Scientists and ML Engineers applying advanced techniques to build or assess retrieval-augmented generation (RAG) systems.

Model Risk and Validation Professionals ensuring AI systems meet regulatory and internal governance standards.

Portfolio and Market Risk Analysts evaluating how AI-related risks can affect financial exposure.

Regulators, Consultants, and Solution Providers supporting compliance, oversight, or advisory roles for AI and model governance in financial

For professionals in AI, finance, and risk tasked with building, validating, or governing AI systems

Course outcomes and benefits:

Explore emerging industry trends in managing the risks of generative AI, including shifting regulatory expectations and evolving best practices for validation and governance.

Industry Trends in Risk Management of Generative AI

Learn a structured approach to validating RAG-based generative AI systems, grounded in information retrieval theory, text embedding analysis, broad test coverage, and human calibration—designed to ensure outputs are relevant, grounded, and aligned with organizational goals.

A Formal Validation Approach for RAG-based Generative AI Systems

Gain practical insights into red-teaming and field testing for generative AI, including techniques to simulate real-world misuse, identify vulnerabilities, and assess system behavior under stress—helping ensure resilience, safety, and compliance in dynamic deployment settings.

Red-teaming and Field Testing  Approaches for Generative AI

Understand key themes of NIST AI 600-1, the Generative AI Risk Profile, including risks tied to hallucination, prompt injection, data provenance, and human-AI interaction. Learn how these themes map to broader AI governance goals like transparency, fairness, and accountability.

Themes of NIST AI 600-1

<ul><li>Enjoy a live, interactive course without the cost or hassle of travel.</li><li>Access course materials in advance and anytime via the event portal.</li><li>Learn through real-world examples and case-based discussions.</li><li>Join live conversations with peers and request recordings if needed.</li></ul>

Participants will also benefit from:

Evals Are Almost Never All You Need

Principal Scientist, HallResearch.ai

Understand how task contamination, conflicts of interest, and lack of scientific measurement can compromise evals.

Common problems with benchmarks and evals

Learn about embedding-based approaches, red-teaming, and field testing for evaluating generative AI systems.

What to do instead of benchmarks and evals

Developers need benchmarks. They're an important tool for development. They're the wrong tool for real-world assessment.

When to use benchmarks and evals

GenAI Validation - Free Discussion and Q&A Session

VP at H2o.ai (ex-Wells Fargo, B of A, and Ford Analytics Leader)

Principal Scientist at HallResearch.ai (Teacher, Author, Inventor, Advisor)

We'll have some questions for each other, then we'll open up the discussion for audience Q&A.

Meet Agus and Patrick

No stupid questions here, and we'll do our best to answer your difficult questions too.

Bring you simplest or hardest questions

We're offering a course on generative AI validation, July 22 - 24, 2025. We want to hear what potential students think!

Learn directly from Hall Research AI

Hall Research AI

Courses

ASRC AI Ethics Technical Training Course

Explainable AI for Decision-Making Applications

Governing GenAI - From Prompts to Policy

Lightning Lessons

Evals Are Almost Never All You Need

GenAI Validation - Free Discussion and Q&A Session

Understand SHAP (SHapley Additive exPlanations)

Understand SHAP (SHapley Additive exPlanations)