Faithfulness, answer correctness, and citation quality

Overview

• Faithfulness asks whether the answer is supported by supplied evidence.
• Correctness asks whether it is right for the task and real applicable facts.
• Citation quality checks whether references point to supporting sources, not merely existing documents.

Why this matters

An answer can faithfully quote an expired policy or correctly state a general fact without answering the customer's actual policy question.

An everyday example

An employee can accurately repeat an old handbook but still give the wrong current instruction.

A technical example

Score each material claim for evidence support, policy applicability, missing exceptions, and valid source reference.

How it works

1. Break answer into checkable claims
2. Map to eligible evidence
3. Compare with reference outcome
4. Grade completeness/applicability
5. Inspect unsupported statements.

Architecture and flow

Answer + evidence + expected outcome → support/correctness/citation checks.

The insurance assistant example

“Flood damage is covered” fails if the customer's active endorsement excludes it, even if a generic brochure says otherwise.

When to use it

RAG, policy explanations, summaries, and extracted decision-support material.

When to avoid it

Do not use exact string matching as the only grader for open-ended answers.

Interview answer

“Faithfulness checks support from context; correctness checks the applicable answer. I also evaluate source validity, missing conditions, and abstention.”

Benefits, trade-offs, and production considerations

• Retrieval evaluation and answer evaluation need shared source-version fixtures.
• A checker trained only on easy grounded examples can miss subtle contradictions and numerical errors.