Frequently Asked Questions (And Answers) About AI Evals
Hamel Husain and Shreya Shankar, creators of the world’s leading AI Evals course, have distilled answers to the most common questions they’ve received while teaching over 700 engineers and product managers. Their resource covers practical challenges in evaluating large language models (LLMs) and retrieval-augmented generation (RAG) systems, including model selection, annotation tools, evaluation metrics, error analysis, and more. They share dozens of free videos, blog posts, and resources to address widespread misconceptions and provide actionable advice. Their approach emphasizes hands-on, tool-agnostic methods, prioritizing error analysis and customized interfaces for reviewing AI outputs. The FAQ is designed to help teams build more reliable, production-ready AI systems by focusing on evaluation as a core part of the development process, not just a final checklist
1. Frequently Asked Questions (And Answers) About AI Evals
Hamel Husain and Shreya Shankar who run the world’s No. 1 AI Evals course by any metric, answer the most common questions they received while teaching over 700 students.
And share dozens of free videos, blog posts, and resources addressing common misconceptions.
Contents:
- Q: What are LLM Evals?
- Q: Is RAG dead?
- Q: Can I use the same model for both the main task and evaluation?
- Q: How much time should I spend on model selection?
- Q: Should I build a custom annotation tool or use something off-the-shelf?
- Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?
- Q: How do I debug multi-turn conversation traces?
- Q: Should I build automated evaluators for every failure mode I find?
- Q: How many people should annotate my LLM outputs?
- Q: What gaps in eval tooling should I be prepared to fill myself?
- Q: What is the best approach for generating synthetic data?
- Q: How do I approach evaluation when my system handles diverse user queries?
- Q: How do I choose the right chunk size for my document processing tasks?
- Q: How should I approach evaluating my RAG system?
- Q: What makes a good custom interface for reviewing LLM outputs?
- Q: How much of my development budget should I allocate to evals?
- Q: Why is “error analysis” so important in LLM evals, and how is it performed?
- Q: What’s the difference between guardrails & evaluators?
- Q: What’s a minimum viable evaluation setup?
- Q: How do I evaluate agentic workflows?
- Q: Seriously Hamel. Stop the bullshit. What’s your favorite eval vendor?
- Q: How are evaluations used differently in CI/CD vs. monitoring production?
- Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?
- Q: Should I use “ready-to-use” evaluation metrics?
- Q: How can I efficiently sample production traces for review?
Hope that helps!