Evaluation and observability
Practice this questionWANT TO UNDERSTAND THE CONCEPT FIRST?Learn Evaluation and observability
| Measure | Meaning |
|---|---|
| Precision@k | Relevant returned evidence / returned evidence |
| Recall@k | Retrieved relevant evidence / labeled relevant evidence |
| Faithfulness | Answer is supported by supplied evidence |
| Correctness | Answer is right for the applicable task/facts |
| Citation quality | Referenced source actually supports the claim and is eligible |
| Tool accuracy | Correct allowed tool and valid arguments |
| Task completion | Durable intended outcome occurred safely |
| LLM judge | Scalable rubric scoring calibrated against humans |
| Trace | Connected timed model/tool/retrieval/service spans |
| Audit | Durable record of actor, action, authority, and business outcome |
- Evaluate no evidence, stale policies, malformed documents, ambiguous dates, denied access, injected content, restarts, duplicate submissions, and uncertain writes.
- Track source/model/prompt/graph versions, p95 latency, tokens, cost per successful task, queue/review age, and unresolved outcomes.
- Avoid raw private payload logging by default.
Adapted from 13_AI.MD · “Evaluation and observability”. Code examples are study fragments. Difficulty and core labels are editorial estimates; version-specific behavior should be checked against your project.