Reinforcement Learning

Overview

RL is one of the console's seven tabs (DEBUG · MANUAL · COLLECT · REPLAY · RL · EVAL · RESULT). It combines policy rollout and data collection: the policy controls the robot by default, a human can take over selected segments, and the complete episode is saved for training and review.

Prerequisites

• A live robot config with working transport, camera observations, and URDF state.
• A reachable policy server matching the observation and action schema.
• A robot integration that reports HIL support if human intervention is required.
• An RL config under configs/04_rl/; machine-specific values belong in an ignored .local.py override.

Configure and launch

Start from the template for your robot. The policy entry is required; critics may be empty. Rollouts are currently written in LeRobot format.

_base_ = ['r1lite_rl.py']

rl_cfg = dict(
    tasks=['assemble the phone'],
    policies=[dict(
        name='r1lite_policy',
        config='../01_deploy/r1lite/openpi_qpos.local.py',
        host='127.0.0.1',
        port=9000,
    )],
    critics=[dict(
        name='r1lite_critic',
        type='websocket',
        host='127.0.0.1',
        port=9100,
        backend_options={},
    )],
    data=dict(storage=dict(
        log_dir='work_dirs/rl/r1lite',
    )),
)

eva --config configs/04_rl/r1lite_rl.local.py

Workflow

Human-in-the-loop intervention

Turn on HIL before the rollout. While the policy is running, press INTERVENE to hand control to the operator. The source indicator changes immediately and the saved episode marks the intervention interval separately from policy-controlled frames.

• ACCEPT keeps the intervention segment and returns control to the policy.
• ABANDON rejects the active intervention segment.
• Saving is blocked while an intervention is unresolved, so every stored segment has an explicit source.

Critic value

With a critic connected, the CRITIC VALUE panel updates during live rollout and replay. Time in seconds is shown on the horizontal axis. The line reports the critic output, while the chart background distinguishes policy-controlled and intervention-controlled regions.

Starting another episode replay clears the previous value series before the new curve is drawn. Without a critic, the panel stays hidden and the live camera layout uses the full workspace.

Saved data, replay, and QC

The SAVED DATA panel shows save-queue state, progress, ETA, and one tile per episode. Select any completed tile to load it immediately. Replay keeps all camera streams and the URDF on the same timeline; action/state charts appear in replay, while critic values are recomputed when a critic is selected.

Use PASS, FAIL, and the note field for quality control. The selected episode reflects the persisted verdict. Press RETURN TO LIVE to leave review mode and recover the live robot view.

Common states and faults