Evaluate AI behavior with comparable evidence

Record that a steer was consumed. Write-once and idempotent.

run_id required

Positional argument. Eval run id.

steer_id required

Positional argument. Steer id to acknowledge.

--consumed-by required

Who consumed it — an agent session id, not a user id.

--note

What was DONE, e.g. 'applied as derive on run X'.

--json

Emit raw JSON output.

Score a PEFT LoRA adapter directory on held-out retrieval (vs base model).

--family, -f required

Base model family: qwen3-4b, qwen3-0.6b, or mistral-7b.

--adapter-dir, -a required

Adapter directory with PEFT LoRA weights to score.

--examples

Held-out examples to score.

--seed

Seed for reproducible example generation.

--device

auto, metal, cuda, or cpu.

--model-dir

Optional base-model directory override.

--endpoint

HybrIE runtime URL (bypasses the console proxy).

--json

Emit raw JSON output.

Emit lineage identifiers, fingerprints, counts, and outcomes only.

run_id required

Positional argument. Explicit eval run id.

Emit bounded machine-readable status without raw eval or steer content.

run_id required

Positional argument. Explicit eval run id.

List promoted champions, newest promotion first.

--comparability-key

Only the champion(s) of one comparability bucket.

--limit

See the installed command’s --help for accepted values.

--json

Emit raw JSON output.

Create a durable prompt/data/inference eval run.

--name, -n

Eval suite display name.

--data-asset-id

Staged data asset or trace snapshot id. Repeat to use one exact multi-case cohort.

--prompt

Prompt ref as KEY, KEY:VERSION, or KEY:LABEL. Can be repeated.

--provider

Baseline provider.

--model

Baseline model or endpoint model name.

--adapter-id

Optional adapter id to hot-swap as a candidate.

--adapter-version

Exact registered adapter version.

--base-model-id

Exact base model recorded by the adapter manifest.

--adapter-format

Registered adapter format, for example peft-lora.

--evaluator-id

Attach an evaluator. Its definition governs scoring, and its invariants gate every candidate BEFORE any judge spend (see evaluator-create).

--execute

Create the run and start it now.

--leave-queued

Create the run QUEUED and do NOT start it. You start it yourself with execute-run.

--json

Emit raw JSON output.

Archive (default) or hard-delete eval runs.

run_ids required

Positional argument. One or more eval run ids.

--hard

Destroy the rows instead of archiving. Gated on lineage.

--include-descendants

With --hard, also destroy every run branched below the selection.

--yes, -y

Skip the confirmation prompt.

--json

Emit raw JSON output.

Branch a child run from one candidate of a parent run.

run_id required

Positional argument. Parent run id to branch from.

--rationale required

The hypothesis this branch tests. Required — a branch with no stated hypothesis is a rerun.

--prompt-file

File holding the new prompt content. Pushed as a new version of the SOURCE arm's prompt key.

--prompt-ref

Existing prompt as KEY, KEY:VERSION or KEY:LABEL. Its key must match the source arm's.

--kind

Derive kind. adapter_warm_start is the only unavailable training operation.

--source-candidate-key

Branch from this arm instead of the parent's best-scoring one.

--adapter-id

Adapter id for an adapter_hot_swap derive.

--adapter-version

Exact adapter version for an adapter_hot_swap derive.

--base-model-id

Exact base model recorded by the adapter manifest.

--adapter-format

Adapter format for an adapter_hot_swap derive.

--candidate-provider

Explicit provider for a model_substitution or prompt_and_model derive.

--candidate-model

Explicit model for a model_substitution or prompt_and_model derive.

--instruction

Immutable provenance: what this child was steered with. Usually a steer body.

--allow-rejected

Override the rejection ledger: derive even though this exact candidate (same content, key, provider, model and route) was already rejected in this measurement context. Without it the API refuses with a 409.

--stop-parent

Skip the parent's PENDING results. PERMANENT: the parent becomes a partial measurement.

--start/--no-start

Queue and spawn the child now (default), or leave it DRAFT.

--idempotency-key

Retry-safe key. A repeat under the same key returns the first child and spends nothing.

--max-cases

Cap cases drained by the first execution.

--max-candidates

Cap candidates drained by the first execution.

--json

Emit raw JSON output.

Create a reusable evaluator whose invariants gate candidates pre-spend.

--name, -n required

Evaluator display name.

--forbidden-phrase

A candidate whose prompt contains this exact text is SKIPPED before any judge spend and recorded as a rejection. Repeatable.

--required-phrase

A candidate whose prompt is MISSING this text is skipped. Repeatable.

--max-prompt-chars

Skip a candidate whose prompt exceeds this length.

--invariant-key

Identifier recorded on rejections this invariant produces.

--description

Human note stored with the evaluator.

--json

Emit raw JSON output.

List evaluators visible to the active workspace.

--include-archived

Include archived evaluators.

--json

Emit raw JSON output.

Execute a queued durable Lab eval run.

run_id required

Positional argument. Eval run id.

--json

Emit raw JSON output.

Show one durable Lab eval run: best arm, lineage, review and pending work.

run_id required

Positional argument. Eval run id.

--json

Emit raw JSON output.

Score a trained adapter checkpoint on held-out NIAH retrieval (vs zero-LoRA baseline).

--family, -f required

Base model family: qwen3-4b, qwen3-0.6b, or mistral-7b.

--checkpoint-dir, -c required

Trained checkpoint directory (metadata.json + model.safetensors).

--examples

Held-out examples to score (default 200).

--seed

Seed for reproducible example generation.

--device

auto, metal, cuda, or cpu.

--model-dir

Optional base-model directory override.

--endpoint

HybrIE runtime URL (bypasses the console proxy).

--json

Emit raw JSON output.

Apply a promotion proposal: move the production label and pin the champion.

proposal_id required

Positional argument. Proposal id to apply (see proposals).

--reason

Note recorded with the promotion.

--yes, -y

Skip the confirmation prompt.

--json

Emit raw JSON output.

Prepare a proposal for human review without applying or promoting it.

run_id required

Positional argument. Completed eval run id whose winner should be reviewed.

Emit one bounded proposal action without narratives or arbitrary payloads.

proposal_id required

Positional argument. Explicit proposal id to review.

List promotion proposals — a completed run's recommended label move.

--status

Filter by status (proposed, applied, dismissed).

--limit

See the installed command’s --help for accepted values.

--json

Emit raw JSON output.

List the rejection ledger, newest first.

--comparability-key

Only rejections in one comparability bucket.

--identity-hash

Only rejections of one candidate identity.

--run

Only rejections recorded on one run.

--limit

See the installed command’s --help for accepted values.

--json

Emit raw JSON output.

Score a policy on an RL environment and report reward statistics.

--family, -f required

Base model family: qwen3-4b, qwen3-0.6b, or mistral-7b.

--environment

RL environment (e.g. niah).

--policy

Policy to score: base, hypernet, or peft-lora.

--checkpoint-dir, -c

Trained checkpoint directory (required for non-base policies).

--tasks

Number of eval tasks (sent as num_tasks).

--max-new-tokens

Generation cap per task.

--temperature

Sampling temperature.

--seed

Seed for reproducible task generation.

--pass-threshold

Reward threshold above which a task counts as passed.

--device

auto, metal, cuda, or cpu.

--endpoint

HybrIE runtime URL (bypasses the console proxy).

--json

Emit raw JSON output (includes the per_task list).

List durable Lab eval runs, ranked by their best ARM rather than a run mean.

--limit

See the installed command’s --help for accepted values.

--status

Filter by draft, queued, running, completed, or failed.

--include-archived

Also list archived runs (hidden by default).

--json

Emit raw JSON output.

Attach an instruction to a run for whichever agent polls it next.

run_id required

Positional argument. Eval run id.

--body required

The instruction, in plain text.

--kind

instruction or note.

--idempotency-key

Retry-safe key; a repeat returns the existing steer instead of a second one.

--json

Emit raw JSON output.

List a run's steers, newest first, with each consumption note.

run_id required

Positional argument. Eval run id.

--pending

Only steers nobody has consumed yet.

--include-archived

Allow reading an archived run's steers.

--json

Emit raw JSON output.

Show the whole lineage tree containing a run: nodes, best-per-bucket, pending work.

run_id required

Positional argument. Any run id in the tree — root, leaf or middle.

--include-archived

Include archived member runs.

--json

Emit raw JSON output.

list_runs(limit, status, include_archived)

Returns the platform response; inspect its status and error detail.

get_run(run_id)

Returns the platform response; inspect its status and error detail.

get_agent_status(run_id)

Bounded run state without payloads, narratives, steer text, or raw errors.

get_agent_iteration(run_id)

Typed lineage IDs, counts, fingerprints and outcomes only.

list_champions(comparability_key, limit)

Promoted champions, newest promoted_at first.

list_rejections(comparability_key, identity_hash, run_id, limit)

The rejection ledger, newest first.

list_evaluators(include_archived)

Evaluators visible to the active workspace.

create_evaluator(name, forbidden_phrases, required_phrases, max_prompt_chars, invariant_key, description)

Create a reusable evaluator whose invariants gate candidates pre-spend.

list_proposals(status, limit)

Promotion proposals from completed runs, newest first.

review_proposal(proposal_id)

Review one explicit proposal through the bounded agent projection.

prepare_promotion_proposal(run_id)

Prepare a human-review proposal for one completed eval; never apply it.

apply_proposal(proposal_id, reason)

Apply a promotion proposal: move the production label, pin the champion.

create_run(tags)

Create an eval run. A faithful mapping of POST /runs.

execute_run(run_id, max_cases, max_candidates)

Returns the platform response; inspect its status and error detail.

run_url(run_id, view)

Console deep link for a run, or None when the console origin is unknown.

get_run_tree(run_id, include_archived)

The whole lineage tree containing run_id, from any run id in it.

derive_run(run_id, rationale, derive_kind, prompt_ref, prompt_content, source_candidate_key, adapter_id, adapter_format, derive_instruction, stop_parent, start, idempotency_key, max_cases, max_candidates)

Branch a child run from one candidate of run_id.

create_steer(run_id, body, kind, idempotency_key)

Attach an instruction to a run for whichever agent polls it next.

list_steers(run_id, pending, include_archived)

Returns the platform response; inspect its status and error detail.

ack_steer(run_id, steer_id, consumed_by, note)

Record that a steer was consumed. Write-once and idempotent.

delete_runs(run_ids, mode, include_descendants)

Archive (default) or hard-delete runs.

score_niah(family, checkpoint_dir, examples, seed, device, model_dir, timeout)

Score a trained checkpoint on held-out NIAH retrieval (vs zero-LoRA).

score_adapter(family, adapter_dir, examples, seed, device, model_dir, timeout)

Score a PEFT LoRA adapter directory on held-out retrieval (vs base).

score_rl(family, environment, policy, checkpoint_dir, tasks, max_new_tokens, temperature, seed, pass_threshold, device, timeout)

Score a policy on an RL environment and report reward statistics.