EVAL

Checkpoint sweep on the real robot: each task prompt runs N times per checkpoint and is graded by milestone clicks.

Prerequisites

One policy server per checkpoint, already running on its declared host/port. EVAL connects; it does not spawn them. A missing server leaves RUN with no motion. For OpenPI see its policy-server guide; other backends have their own.
Robot powered and connected, or cli_mode='sim'.
An evaluation .py config listing the checkpoints and tasks — see Config for the EVAL.

Launching EVAL

The EVAL and RESULT tabs appear iff the launched config contains a non-empty eval_cfg. Plain deploy configs have neither.

bash

eva --config configs/03_evaluation/arx_r5_eval.py --web-port 8080

Open http://localhost:8080 and click EVAL. The console connects Model A's policy server immediately; the reset-to-start and warm-up are deferred until the first RUN.

The MODEL picker

Each checkpoint becomes one button at the top, labelled Model A, Model B, … alongside the real checkpoint name; order is optionally shuffled via shuffle_ckpts.

Clicking a model button connects its policy, resets the session, and opens a fresh recording so each model's runs are saved separately. Previously saved scores reload. Switching is disabled while a run is in progress.

With a single checkpoint listed, one model button shows. With checkpoints omitted, EVAL runs one unnamed model straight from the base config.

Task instructions & trials

The left rail lists every task instruction (Chinese label if provided, else English) with a row of trial cells per instruction (default five). Clicking an instruction jumps to its first untested trial.

Each tasks entry:

Task field	Meaning
`prompt_en`	English instruction sent to the model. Also the cell identifier — keep it stable.
`prompt_zh`	Optional Chinese label for the rail. Display only.
`milestones`	Ordered `(id, label)` pairs you grade against.
`init_pose`	Optional starting joint angles (radians, joint order). Defaults to the home pose. Copy from a pose you have already commanded rather than guessing.

Running and scoring a trial

Trial loop:

1. Pick a Model button.
2. Pick an instruction and a trial cell.
3. RUN ▶ — arm resets, model warms up, attempt begins.
4. The robot-state badge shows the stage: resetting… → warming up… → validating policy… → running.
5. STOP ■ closes and saves the clip immediately (or it auto-stops).
6. Click milestones, then SAVE SCORE · NEXT.

RESET ⟲ re-runs the reset-to-start at any time. With reset_after_each_trial=True, STOP resets automatically; otherwise the next RUN resets just before it starts.

Warm-up. The first attempt loads and validates the model against a test observation — non-trivial latency, not a hang; watch the badge. Later trials skip it (skip_warmup_after_first).

Milestone scoring

Scoring is a ladder, not a free checklist: milestones are ranked, click the highest one reached. Everything below lights up automatically; everything above clears. Click the top one again to undo.

score = rungs reached, max = total milestones. A trial counts as success only when score equals max.

SAVE SCORE · NEXT writes the grade into the recorded clip (no separate results file) and advances. The cell shows its score afterwards; a blank cell means the save did not stick.

Scoring requires a recorded clip. Scoring an unrun trial is refused with run it first. Score controls also lock during a run.

Re-running a trial records a fresh, unscored clip. The grid shows one record per cell and prefers the scored one, so a re-run never hides a saved score.

Reviewing a trial & the INIT panel

The right column shows the live arm and cameras by default. Clicking a trial that already has a clip flips it to replay: camera videos, 3D arm view, and joint charts play back on one timeline. Same player as RESULT, where per-model and side-by-side score tables live.

An INIT panel under EVAL drives the arm to a chosen start pose before a run. Without init_pose, the start pose is home.

Serving a model on another machine (optional)

If a policy server runs remotely, EVAL can open an SSH tunnel per distinct model port. Configure via:

Key	Value
`enable_ssh_forward`	`True` to open the tunnel(s); one per distinct model port.
`ssh.host`	Remote machine address.
`ssh.user`	Remote login user.
`ssh.port`	Remote SSH port (usually `22`) — not the policy-server port.
`ssh.remote_sync_dir`	Optional. If set, results are pulled back from this remote directory.

Key-based SSH login must already work. If the tunnel fails to open, the policy connection fails as if the server were unreachable — verify ssh by hand first.

Config for the EVAL

Everything above is driven by an eval_cfg section in a Python .py config. An evaluation config inherits a deploy config (robot, cameras, transport, policy) via _base_ and adds eval_cfg with the checkpoint and task lists.

Copy an existing file such as configs/03_evaluation/arx_r5_eval.py and edit the marked values; the skeleton below shows the shape.

configs/03_evaluation/my_eval.py

# Inherit robot + policy from a deploy config.
_base_ = ['<路径/deploy 配置.py>']   # ← your deploy config

eval_cfg = dict(
    trials_per_prompt=5,            # repeats per instruction
    cli_mode='real',             # 'real' or 'sim'
    inference_strategy='async',    # keep 'async' unless you know otherwise

    checkpoints=[                  # one entry per model
        dict(
            name='<模型名>',            # ← label shown next to the Model A/B slot letter
            config='<路径/deploy 配置.py>',  # ← this model's deploy config
            port=<端口>,              # ← its policy-server port
        ),
        # ... more dict(...) for more models ...
    ],

    tasks=[                        # one entry per instruction
        dict(
            prompt_en='<英文指令>',       # ← instruction to the model
            milestones=(                  # graded steps, easiest first
                ('approach', 'approach the object'),
                ('pick',     'pick it up'),
                ('place',    'place it down'),
            ),
        ),
    ],
)

A non-empty eval_cfg is what enables EVAL/RESULT. An empty section means "no evaluation" and both tabs disappear.

Every `eval_cfg` field

Per-instruction fields are in Task instructions & trials.

Field	Default	Meaning
`tasks`	`[]`	Instructions to test. Each: `{prompt_en, prompt_zh?, milestones?, init_pose?}`. At least one required.
`trials_per_prompt`	`5`	Repeats of each instruction, per model.
`checkpoints`	`[]`	Models in the checkpoint sweep. Omit to run one unnamed model off the base config.
`cli_mode`	`'real'`	`'real'` or `'sim'`.
`inference_strategy`	`'async'`	One of the names in inference strategies.
`reset_after_each_trial`	`False`	`True`: STOP resets automatically; `False`: next RUN resets before starting.
`skip_warmup_after_first`	`True`	`True`: only the first trial warms up.
`storage`	`dict(fps, save_queue_max)`	`fps` = saved frame rate; `save_queue_max` caps write-queue depth.
`shuffle_ckpts`	`False`	Randomize the trial order across checkpoints.
`shuffle_seed`	`42`	Seed for the shuffle, for reproducibility.
`enable_ssh_forward`	`False`	Open one SSH tunnel per model port; requires `ssh` (see Serving a model on another machine).
`ssh`	`—`	`dict(host, user, port, remote_sync_dir?)`; see that section.

Each checkpoints entry:

Checkpoint field	Meaning
`name`	The checkpoint's real name; shown as a slot letter (Model A / B / …) plus this name in the picker, and used in saved records.
`config`	Path to this model's deploy config.
`port`	Policy-server port.
`host`	Optional, defaults to `127.0.0.1`.

Robot, transport, policy, and action-space settings come from the inherited deploy config — see Configuration, Transport, Backends, Action spaces.