Production Trace Triage (QA)

Production Trace Triage (QA)

1. 01

   Define the failure taxonomy before opening any trace viewer

   Agree on four to six labels with the client (hallucinated fact, wrong tool call, retrieval miss, latency breach, tone drift) so triage produces comparable counts week over week rather than one-off anecdotes.

2. 02

   Pull a stratified sample of 50 to 100 production sessions per client workflow

   Stratify by intent and by model tier, since a cheaper tier handling 80 percent of traffic will dominate any unweighted sample and hide failures concentrated in the long tail.

3. 03

   Replay each flagged trace end to end, including tool calls and retrieval steps

   Platforms such as Langfuse and Arize capture hierarchical spans, so the reviewer can see whether the model reasoned badly or simply received bad context from the retrieval layer.

4. 04

   Score outputs against the agreed rubric using a calibrated judge

   Tune the LLM-as-judge prompt against 20 human-labeled examples first; Braintrust and Confident AI both support judge calibration, and an uncalibrated judge will drift toward passing everything.

5. 05

   Separate cost and latency defects from quality defects in the report

   Token waste and slow spans are fixable by prompt compression or caching, while quality defects need rubric or model changes; mixing them in one backlog delays the cheap wins.

6. 06

   File each confirmed defect with a reproduction trace ID and an owner

   A defect without a replayable trace ID gets re-debated in the next review; attach the session identifier and the exact input that triggered it.

7. 07

   Re-run the same sample slice after the fix and compare pass rates

   Hold the sample constant for two cycles so the client sees movement on a like-for-like basis instead of a fresh baseline that always looks improved.