Skip to content
Instant download · PDF + web edition · Free sample of every book · 30-day refund promise
Latent Library Browse books
AI Engineering

Evaluating AI Systems

Metrics, Test Sets and Judgement for Models That Matter

Pages
81
Chapters
18
Level
Intermediate–Advanced
Format
E-book

About this book

'It looked good in the demo' is not an evaluation. Evaluating AI Systems shows how to measure whether a model or AI feature truly works for its users, and how to keep measuring once it is live.

The book covers classic metrics (accuracy, precision, recall, F1, calibration, error analysis) with worked examples; how to build test sets that reflect real use; methods for evaluating generated text, including rubrics, pairwise comparison and human review; when and how to use a language model as a judge; online evaluation with A/B tests and monitoring; and red teaming for safety and robustness.

It ends with an evaluation plan template that you can adapt for any AI project, from a classifier to a chat assistant.

What you will learn

  • Choosing metrics that match real costs and benefits
  • Precision, recall, F1, calibration and error analysis with examples
  • Building representative, versioned test sets
  • Evaluating generative output with rubrics and pairwise comparison
  • Using LLM judges reliably, and checking the judges
  • A/B testing, monitoring and red teaming in production
  • Confidence intervals, paired comparisons, the bootstrap and sample sizes, with numbers
  • How to evaluate RAG systems, agents, fairness, speech and vision
  • How to run human evaluation, build an evaluation pipeline and decide by expected cost

Table of contents

  1. IntroductionIntroduction
  2. Chapter 1Why Evaluation Is the Real Work
  3. Chapter 2Classic Metrics, Worked Through
  4. Chapter 3Statistics for Evaluators
  5. Chapter 4Building a Test Set
  6. Chapter 5Evaluating Generated Output
  7. Chapter 6Language Models as Judges
  8. Chapter 7Evaluating Retrieval and RAG Systems
  9. Chapter 8Evaluating Agents and Multi-Step Systems
  10. Chapter 9Online Evaluation and Monitoring
  11. Chapter 10Fairness Evaluation
  12. Chapter 11Red Teaming and Robustness
  13. Chapter 12Speech, Vision and Multimodal Systems
  14. Chapter 13Reading Benchmarks and Leaderboards Critically
  15. Chapter 14Case Study: Evaluating a Support Assistant
  16. Chapter 15Running Human Evaluation Well
  17. Chapter 16Building an Evaluation Pipeline
  18. Chapter 17From Metrics to Decisions: Expected Cost
  19. Chapter 18An Evaluation Plan Template
  20. AppendixExercises
  21. GlossaryGlossary

Who this book is for

Data scientists, ML and AI engineers, product managers and QA professionals responsible for AI quality. Basic familiarity with machine learning is assumed.

Details

FormatPDF + web edition
Length81 pages (approx.)
Chapters18
LevelIntermediate to Advanced
LanguageEnglish
TopicAI Engineering
AuthorLatent Library Editorial

Keep reading

You may also like

All books
$11.99
Details