Evaluation

Durable eval runs#

Lab evaluations are first-class workspace resources. A run persists the suite, cases, candidates, pending results, execution routing, and lineage before compute starts. The same route covers prompt regression, curated data or trace snapshots, baseline model endpoints, hot-swapped adapters, and external endpoint candidates.

From Python, client.lab_evals wraps these endpoints (see the Python SDK):

from stimulir import StimulirClient

client = StimulirClient()
run = client.lab_evals.create_run(
    name="Customer Agent prompt regression",
    data_asset_id="<snapshot-id>",
    prompt_ref="customer.chat.agent:prod",
    model="hybrie-runtime-default",
    execute=True,
)
report = client.lab_evals.get_run(run["id"])

# Create a durable prompt/data eval run
stimulir lab eval create-run \
  --name "Prompt and data regression" \
  --data-asset-id <asset-id> \
  --prompt agent_chat_browser_research:staging \
  --model hybrie-runtime-default

# Create and execute in one call
stimulir lab eval create-run --data-asset-id <asset-id> --prompt <key>:<version> --execute

# List and execute runs
stimulir lab eval runs
stimulir lab eval get <run-id>
stimulir lab eval execute-run <run-id>

curl -X POST https://api.stimulir.com/api/v1/lab/evals/runs \
  -H "Authorization: Bearer $STIMULIR_TOKEN" \
  -H "X-Business-Profile-Id: $WORKSPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "suite_name": "Prompt and data regression",
    "source": "mixed",
    "data_asset_id": "<asset-id>",
    "prompt_refs": [{"key": "agent_chat_browser_research", "label": "staging"}],
    "baseline_provider": "hybrie",
    "baseline_model": "hybrie-runtime-default",
    "execute": true
  }'

PEFT adapter evaluation#

Scores a classical PEFT adapter, produced by SFT training, an RL run with policy=peft-lora, exported from the D2L pipeline, or registered externally, on held-out Needle-In-A-Haystack retrieval, against the zero-LoRA baseline. The call is synchronous.

The metrics are nested under a report key in the response. It carries four accuracy metrics, n, adapter_acc, base_acc, lift (adapter_acc − base_acc), plus the adapter's configuration so the score is self-describing:

curl -X POST http://localhost:8080/v1/eval/adapter \
  -H "Content-Type: application/json" \
  -d '{
    "family": "qwen3-4b",
    "adapter_dir": "~/hybrie-mounts/d2l-artifacts/rl-1718000000/adapter",
    "examples": 200,
    "seed": 42
  }'

stimulir lab eval adapter --family qwen3-4b --adapter-dir <dir>

Reward evaluation (policies)#

Reward evaluation of a policy on a verifiable environment: the harness samples tasks from the environment, generates completions with the chosen policy, and scores each completion with the environment's reward function. The call is synchronous.

The report summarizes the reward distribution and per-task outcomes:

To measure lift, run the eval twice with the same environment, seed, and num_tasks, once with policy=base for the floor, then with your trained policy, and compare mean_reward and pass_rate:

# Baseline
curl -X POST http://localhost:8080/v1/eval/rl \
  -H "Content-Type: application/json" \
  -d '{"family": "qwen3-4b", "policy": "base", "num_tasks": 50, "seed": 42}'

# Trained policy
curl -X POST http://localhost:8080/v1/eval/rl \
  -H "Content-Type: application/json" \
  -d '{
    "family": "qwen3-4b",
    "policy": "peft-lora",
    "checkpoint_dir": "~/hybrie-mounts/d2l-artifacts/rl-1718000000/adapter",
    "num_tasks": 50,
    "seed": 42
  }'

stimulir lab eval rl --family qwen3-4b --policy peft-lora --checkpoint-dir <dir> --tasks 50

Hypernet checkpoint evaluation (NIAH)#

Scores a trained hypernet checkpoint, the artifact produced by Doc-to-LoRA hypernetwork training, on held-out Needle-In-A-Haystack retrieval: known facts ("needles") are buried in long filler context, and the model must retrieve them exactly. The harness runs the model twice, with the trained checkpoint and with no adapter (zero-LoRA baseline), so the report isolates what the training actually contributed. The call is synchronous.

The metrics are nested under a report key in the response:

curl -X POST http://localhost:8080/v1/eval/niah \
  -H "Content-Type: application/json" \
  -d '{
    "family": "qwen3-4b",
    "checkpoint_dir": "~/hybrie-mounts/d2l-artifacts/train-1718000000/checkpoint",
    "examples": 200,
    "seed": 42
  }'

stimulir lab eval niah --family qwen3-4b --checkpoint-dir <dir> --examples 200 --seed 42

Automatic post-training eval#

You usually don't need to call the NIAH endpoint by hand: every Doc-to-LoRA training job runs the same held-out eval automatically when it finishes, using a disjoint seed so eval examples never overlap the training data. The resulting report (adapter_acc, base_acc, lift) is embedded in the job's final run report alongside the per-step loss curve, see Training Jobs.

Audio evaluation#

Signal-level quality gates for speech audio: peak/RMS level (dBFS), crest factor, clipping and flat-run detection, plus optional intelligibility checking, a local Whisper transcript and word-error-rate against an expected_text. The response includes a gates object with pass/fail booleans per check.

Evaluators#

When you evaluate a prompt against a dataset, scoring is layered deterministic-first. Verifier-derived checks run before any model is asked, and an LLM judge is invoked only on the residual, the part the deterministic checks cannot rule on. This ordering keeps scoring cheap where the answer is knowable, and it is the primary defense against reward-hacking, because a model judge can never override a deterministic result.

Available now

Coming soon

Two more deterministic evaluators are wired and will activate automatically once the candidate executor captures the evidence they need. They are listed here so the roadmap is explicit. Today they return unscored for real cases rather than a misleading number.

• Tool-trajectoryExperimental Deterministic scoring of the sequence of tool calls a trace made against what it should have made. Activates once candidate-side tool-call capture is added to the executor.
• RAG / retrievalExperimental Deterministic scoring of retrieved-document relevance. Activates once retrieval-span capture is added to the executor.

What is measured today, and what is not#

Evaluation has two layers. The durable Lab route records prompt, data, endpoint, and adapter comparison runs with lineage and result rows. HybrIE's low-level benchmark endpoints remain deterministic and verifiable: exact-match retrieval for adapters and checkpoints, and reward evaluation of policies with pass-rate gates via /v1/eval/rl. The honest caveat is that niah is the only built-in reward environment today; the runtime's Environment trait is the extensibility hook for additional verifiable environments.