Evaluation

Beginner — Why isn't normal unit testing enough?

• Model outputs vary and can be plausible but wrong.
• I still unit-test deterministic code, but also evaluate representative tasks, support from evidence, tool behavior, and safe failure using versioned datasets and rubrics.

Intermediate — Faithfulness versus correctness?

• Faithfulness asks whether the answer follows its evidence.
• Correctness asks whether it is the right answer for the applicable task and facts.
• An answer can faithfully repeat an outdated or inapplicable policy and remain wrong.

Senior — How do you trust an LLM judge?

• I calibrate it against human-labeled cases, test adversarial outputs, track disagreement, and version its rubric/model.
• I combine it with deterministic checks and never make it the sole permission or financial gate.

Scenario — A new prompt raises average accuracy but causes one tenant leak. Ship?

• No.
• Severe boundary violations are release blockers regardless of the average.
• I investigate the cause, fix the enforced access boundary, add the case to the security suite, and rerun relevant regression and quality checks.