Evaluation tools: LangSmith, Langfuse, and Arize Phoenix
Practice this questionOverview
- These platforms help collect traces, datasets, scores, and experiment results.
- LangSmith integrates closely with the LangChain ecosystem; Langfuse provides AI observability and evaluation workflows; Phoenix supports tracing and evaluation analysis.
- They are tools for implementing evaluation, not definitions of what quality means.
Why this matters
Engineers need reproducible comparisons and a way to inspect failures across runs.
An everyday example
A test laboratory records results, but you still define what passes inspection.
A technical example
Compare prompt versions on a claims dataset and drill into cases where policy evidence was missing.
How it works
- Instrument calls
- Attach version tags
- Upload permitted cases
- Run scoring
- Compare experiments
- Inspect traces and add regression cases.
Architecture and flow
Application/eval runner → traces + dataset + scores → evaluation platform → decision.
The insurance assistant example
The insurer tracks unsupported-policy-answer rate by product and policy version.
When to use it
When trace inspection and experiment management exceed a simple local report.
When to avoid it
Do not upload raw private cases without approved handling or choose a platform before defining the quality rubric.
Interview answer
“Evaluation platforms organize evidence and comparisons. I still own datasets, risk-based metrics, release thresholds, and privacy controls.”
Benefits, trade-offs, and production considerations
- Check hosting, retention, export, sampling, and cost.
- LangSmith, Langfuse, Arize Phoenix.
Example release gate — illustrative targets, not industry guarantees: zero unauthorized writes or tenant leaks in the security suite; all business-rule fixtures pass; retrieval recall and final correctness meet task-specific targets; p95 latency and cost per completed task stay within the agreed budget. Report counts and uncertainty and investigate every severe failure even when averages pass.