Why Scripted Testing Matters

An important lesson is to test the deterministic engine without depending on live LLM variability.

A scripted model can return fixed extraction results, while scripted connectors return predefined API responses. With a controlled clock and stable workflow content, the same scenario can be replayed repeatedly.

Example scenario:

scenario: unrecognized_transaction
turns:
  - user: "I don't recognize a charge."
    extracted_intent: report_unrecognized_transaction
  - user: "It was $249.99."
    extracted_amount: 249.99
expected:
  next_step: ask_for_transaction_identification

The exact test format depends on the engine. The principle is to assert not just the final message, but also:

• state and ledger updates;
• visited nodes and transitions;
• mandatory questions and confirmations;
• connector calls and idempotency behavior;
• audit events;
• termination and failure paths.

Golden scenarios are particularly useful: they preserve known expected behavior and detect unintended changes during refactoring.