Tool-call accuracy and task completion

Overview

• Tool-call accuracy measures whether the right allowed tool and arguments were selected.
• Task completion measures whether the intended business outcome actually occurred.
• Final text claiming success does not establish completion.

Why this matters

Agents can use the wrong tool, repeat writes, or report completion after a timeout.

An everyday example

A clerk saying “filed” matters only if the filing system contains the receipt.

A technical example

A successful claim-submission case requires the correct approved draft, one durable claim, a receipt, and no forbidden calls.

How it works

1. Define allowed/required actions
2. Capture tool sequence
3. Check args and permissions
4. Verify durable outcomes
5. Score completion and failure handling.

Architecture and flow

Agent run trace + business receipts → action checks → task outcome score.

The insurance assistant example

Simulate provider failure during submission and verify that reconciliation avoids duplicate claims.

When to use it

Any tool-enabled or workflow-driven application.

When to avoid it

Do not score only the model's final answer or require one exact tool sequence when several safe sequences are valid.

Interview answer

“I grade agent outcomes against actual service state and action constraints, including correct arguments, approval, idempotency, and safe failures.”

Benefits, trade-offs, and production considerations

Test cancellation, duplicate messages, concurrent resumes, denial, and missing information. Distinguish correct escalation from unsuccessful automation.