Evaluate a change and review the evidence

Updated 12 October 2026
Applies to Console v0.2.2 with the current navigation. Available actions depend on project access and runtime configuration.

Before you start#

You need the intended project, captured request/response content, representative cases, and available candidate and judge routes. Prompt comparisons need a known current prompt and a different candidate. Automatic Improve also needs registered production prompt lineage.

For example, to improve support replies, use a stable support-replies tag, retain cases where the current reply works and fails, and define what a good reply must do. Choose a small setup check first; a one-case result is not evidence for production promotion.

Optionally save the cases first#

1. Open Lab → Datasets → New dataset.
2. Enter a recognisable name, select a Tag, choose Dates, and set Most traces. The current form allows up to 250, newest first.
3. Check the matching count. With no matches, check the project and widen the dates or choose a different tag.
4. Click Create dataset to save the selection. This writes data; it does not add human labels.
5. Inspect the saved cases and the Cleaned and In Lab counts. Use the retained, usable cases as your comparison cohort.

You can also choose a tag and date range directly in the evaluation wizard. It creates or reuses the corresponding dataset when starting. Keep the resulting dataset/snapshot identity with the run so later comparisons use the same evidence.

Follow the four-step evaluation wizard#

1. What to evaluate

Open Lab → New evaluation. Select Prompt to compare the current prompt with a candidate on the same model, or Model to assess the trace model and compare alternative models or an available adapter. Click Continue.

2. Data

Choose Tag and dates or Saved dataset. For a tag, choose the dates and maximum number of traces. Verify the cohort, not just the displayed count. A zero-match selection disables Continue.

Read the privacy notice and open Change in Data policy if the current policy is unsuitable. Starting from tags collects and cleans cases before evaluation; that is not a promise that every matched case survives or is safe to reuse.

3. Compare

For Prompt, inspect Current. It may be a registered version or raw text from a trace. Select a saved candidate or Write a new version, then inspect the diff. An unchanged candidate cannot continue. The current text must be the actual prompt you intend to compare.

For Model, choose Dataset answers to score saved answers without replaying them, or Another model to generate a fresh baseline. Under Compare with (optional), select up to three candidates, or leave it empty to score only the baseline. Judge calls can still incur cost when the saved answers are reused. Keep unrelated settings fixed. A catalogued model still needs permission, an admitted route, and any required compute.

4. Run

Give the run a recognisable Name. Expand Advanced to choose an evaluator suitable for the task and inspect its judge/runtime requirements. Review the summary, cohort, candidates, and available cost information. Start evaluation creates a run and may make multiple candidate and judge requests. Missing estimates do not mean it is free.

Retain the run ID. Open the run from Lab → Evaluations and wait for a terminal result. Do not call a queued or running job complete. For automatic diagnosis instead of a candidate you write, use New prompt evaluation and the Improve guide.

Understand the three privacy controls#

Cleaning recognises specific patterns such as some credential formats, bearer tokens, emails, and IPv4 addresses. It does not guarantee removal of all names, addresses, phone numbers, or sensitive prose. Inspect the resulting cases and permitted data use. “Identifier removal Off” and a later cleaning pass can both be true.

Choose a judge, then check its judgments#

An evaluator defines how to score the task. A judge supplies a judgment where required. HybrIE eval 1 combines Clef or Clef Flash with an independent LLM judge. Its current configuration weights the two scores equally and flags disagreement above 0.25 for review. Select HybrIE eval 1 under Advanced, then choose a Decision model and an LLM judge. Start evaluation stays disabled while the required judge is missing. Both components need runnable routes.

A disagreement is a reason to inspect a case. It does not prove which judge is correct. To measure evaluator accuracy, compare against independently supplied human labels, not another model’s score alone.

Read the result in this order#

1. Status and coverage: how many cases completed with valid judgments? Which were excluded or failed?
2. Comparable setup: were cases, evaluator, and relevant settings the same for every candidate?
3. Case evidence: inspect failures, disagreements, and representative wins. Read the actual outputs.
4. Tradeoffs: compare quality, latency, and measured cost. A missing value shown as – is unknown.
5. Decision: retain the current version, gather better data, iterate, or prepare a reviewed promotion.

Use Lab → Review#

Suggestions waiting contains proposed prompt or model changes. Open the linked evaluation before acting. Applying a prompt suggestion can make it live. For a model suggestion, follow the stated routing change; Mark as done records completion rather than performing that model change for you.

Human review is the case queue. Use Pending, Approved, Corrected, Rejected, plus reason and assignee filters. Sign-offs can become examples for Lab. Review rules, manual flags, and supported evaluation disagreement/error flows determine what enters the queue; not every trace appears automatically.

An empty Pending tab is not proof that every request passed. Open Review rules to inspect Settings → Project → Review. Evaluate the quality of the underlying evidence before approving any live change.

For CLI and SDK operations, continue to Evaluation reference, Data assets, and Versioned prompts.