Skip to content
ML to LLMs
Learning path

Module 17

LLM evaluation

Measure LLM systems layer by layer: task quality, retrieval, grounding, safety, and operations.

  1. 1. Evaluating LLM systemsBuild an evaluation set, measure retrieval with recall@k, MRR, and nDCG, score answers for correctness and groundedness, use LLM judges carefully, track latency and cost, and compare system versions soundly.45 min

You cannot improve what you cannot measure, and LLM outputs are hard to measure. This module builds an evaluation stack from exact-match metrics to rubric grading, retrieval metrics, groundedness checks, LLM-as-judge (and its limits), and operational metrics like latency and cost.

LayerExample metrics and checks
Task qualityaccuracy, F1, exact match, rubric score
Retrievalrecall@k, MRR, nDCG, context relevance
Groundingclaim support, citation correctness, faithfulness
Safetyharmful-content rate, prompt-injection resistance, privacy
Operationsp50/p95 latency, tokens/sec, error rate, cost per request
Human factorsusefulness, calibration, user correction rate
#LessonYou will be able to
1Evaluating LLM systemsdesign a test set, compute retrieval and answer metrics, and run a sound comparison between two system versions
Next moduleModule 18: LLM inference serving