Evaluation
Words made simple
A few useful meanings before you read the explanation.
- LLM
- A model trained to work with language. It can write or summarize text, but its answer still needs checking against reliable information.
- Deterministic
- Following steps or rules defined by code. The model does not decide which step should run next in a deterministic workflow.
- Tenant
- A customer or organization whose data must stay separate from other customers in a shared application.
Beginner — Why isn't normal unit testing enough?
- Model outputs vary and can be plausible but wrong.
- I still unit-test deterministic code, but also evaluate representative tasks, support from evidence, tool behavior, and safe failure using versioned datasets and rubrics.
Intermediate — Faithfulness versus correctness?
- Faithfulness asks whether the answer follows its evidence.
- Correctness asks whether it is the right answer for the applicable task and facts.
- An answer can faithfully repeat an outdated or inapplicable policy and remain wrong.
Senior — How do you trust an LLM judge?
- I calibrate it against human-labeled cases, test adversarial outputs, track disagreement, and version its rubric/model.
- I combine it with deterministic checks and never make it the sole permission or financial gate.
Scenario — A new prompt raises average accuracy but causes one tenant leak. Ship?
- No.
- Severe boundary violations are release blockers regardless of the average.
- I investigate the cause, fix the enforced access boundary, add the case to the security suite, and rerun relevant regression and quality checks.
Can you explain it without the notes?
Evaluation
Use your own words. Explain an example and one limitation before checking the explanation.
Check your explanation
Beginner — Why isn't normal unit testing enough?
Turn this concept into a clear answer.
Practice this conceptAdapted from 13_AI.MD · “29.15 Evaluation”. Examples are study fragments; framework behavior can depend on your version.