PICO-based WebXR VR teleoperation

Positioning and support scope

This page documents the PICO-based WebXR input path added in 0.2.0. The VR node is an external input adapter, not a robot driver. It serves a browser page, reads WebXR controller poses and buttons, validates and normalizes them, and publishes normalized frames to EVA over ZeroMQ. It does not connect to a robot SDK and it does not drive motors directly. EVA's transport remains the component that executes a validated action on the robot.

PICO is the primary and recommended device path. PICO 4 and PICO 4 Ultra controller profiles work with the example node. Quest is compatible with the same node: Oculus Touch profiles use the same normalization and retargeting contract. Device compatibility does not imply universal robot support. Only the three collection presets below expose a WebXR VR client in this repository. The other listed robots continue to use physical Leader/Follower or ROS transport input.

Control data path: controller input to validated robot action

The browser sends a browser frame (browser-frame version is 1) to the node's authenticated /ws route. The node validates the token, pose dimensions, quaternion, and XR reference space. It wraps the result in protocol eva.teleop.vr with version=3, assigns a session identifier, and carries forward the browser's monotonically increasing sequence, then publishes it on ZMQ PUB endpoint tcp://127.0.0.1:8765. EVA's VR client subscribes to that endpoint. Event acknowledgements travel back through ZMQ ACK endpoint tcp://127.0.0.1:8766.

For every control tick, EVA reads the robot's current joint feedback and computes its current canonical EEF state. Each arm is represented by eight values: x y z qw qx qy qz gripper. VrTeleopClient produces canonical EEF targets plus an active-arm mask. The application uses the current qpos as the IK seed, solves for robot joints, runs FK to measure the residual, applies the joint and gripper limits, and calls transport publish_action. The node owns input and operator events; the client owns retargeting; the EVA application owns gates, IK/FK checks, and publishing.

PICO/Quest WebXR
    │  WebSocket /ws (token)
    ▼
vr_webxr/node.py (protocol v3, events, heartbeat)
    │  ZMQ PUB 8765 / PULL ACK 8766
    ▼
EVA VrTeleopClient (relative retargeting, per-arm authorization)
    │  canonical EEF → IK → FK residual and qpos limits
    ▼
EVA transport.publish_action → real robot

Prepare the PICO path

The procedure below uses the recommended topology: EVA and the node run on a remote host, while an Ubuntu host is connected to the PICO over USB. Run relative commands from the eva-client-new repository root. The headset reaches a loopback URL through SSH and ADB forwarding. Do not expose an unencrypted remote-IP HTTP page to the headset.

• EVA environment: create the repository's .venv, make the eva command available, and make sure the node's Python dependencies import successfully.
• Ports: this procedure uses page port 43876, frame PUB port 8765, and acknowledgement PULL port 8766. Check that another node is not already bound to any of them.
• Ubuntu and PICO: install Android Platform Tools so adb is on PATH. Connect the headset by USB and accept its USB debugging authorization prompt.
• Device browser: use a browser that can open the WebXR page and enter MR. Before a controller is tracked, its frame is valid=false; EVA will not use that hand for spatial motion.
• Robot safety: power and connect the robot, know the physical emergency stop, and begin with a safe posture. Verify the axis rotation, workspace, and gripper endpoints before allowing real motion.

Start PICO end to end

Start the WebXR node first, then start EVA with one of the supported collection configs. When --token is omitted, the node generates a random token and prints it in its log. Copy that exact token for the headset. The token is part of the WebSocket authorization contract and cannot be omitted from the PICO launcher.

Start the WebXR node

From the EVA Client repository, run the command in the VR example README with the page and ZMQ ports used by the official presets:

cd "$CLIENT_ROOT"
source .venv/bin/activate

python examples/input_sources/vr_webxr/node.py \
  --host 127.0.0.1 \
  --port 43876 \
  --endpoint tcp://127.0.0.1:8765 \
  --ack-endpoint tcp://127.0.0.1:8766

The repository wrapper can locate the repository and activate its environment for you. It passes VR_TOKEN through to the node:

VR_TOKEN="<OPTIONAL_TOKEN>" ./examples/input_sources/vr_webxr/run_node.sh

An empty token still lets the node generate one, but you must then use the token printed by that node instance. Keep the node terminal open so its WebXR node ready, PICO command, and receive diagnostics remain visible.

Forward page port 43876 over SSH

On the Ubuntu host connected to PICO, keep this SSH tunnel running. Here <remote-host> is the host where the node is listening. The tunnel maps Ubuntu loopback port 43876 to the remote host's loopback port with the same number.

ssh -N \
  -p 22 \
  -o ExitOnForwardFailure=yes \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -L 43876:127.0.0.1:43876 \
  <remote-user>@<remote-host>

Check ADB and open the PICO page

Query the device before launching. If exactly one authorized device is present, the script detects its serial. With multiple devices, pass the PICO serial explicitly.

adb devices -l

export VR_TOKEN="<TOKEN_FROM_NODE_LOG>"
./examples/input_sources/vr_webxr/open_pico.sh

open_pico.sh requires VR_TOKEN, checks that the selected ADB state is exactly device, runs adb reverse tcp:43876 tcp:43876, and opens a URL containing the token, mode=ar, and a reload timestamp. Choose a device either way shown below; a command-line serial takes precedence over PICO_SERIAL:

./examples/input_sources/vr_webxr/open_pico.sh "<PICO_SERIAL>"

export PICO_SERIAL="<PICO_SERIAL>"
export VR_TOKEN="<TOKEN_FROM_NODE_LOG>"
./examples/input_sources/vr_webxr/open_pico.sh

After the page opens, click ENTER MR to enter the WebXR session. To force a browser refresh, the README uses this URL and Android command:

VR_URL="http://127.0.0.1:43876/?token=<TOKEN_FROM_NODE_LOG>&mode=ar&reload=$(date +%s)"
adb -s "$PICO_SERIAL" shell "am start -S -a android.intent.action.VIEW -d '$VR_URL'"

Launch one of the three official collection presets

In another terminal, still at the EVA Client repository root, choose exactly one command. Each preset sets collection.teleop.control_source to "client" and constructs a vr_webxr client. The commands are the complete official WebXR collection entrypoints:

In COLLECT, confirm robot feedback and the VR client connection. Short-release the right B to open the global ARM gate, long-press each hand's grip to authorize the corresponding arm, and short-release the right A to start recording. During a take, leave the relevant authorization on (it is a toggle, not a continuous grip hold). Short-release right A again to stop and save; hold right A for the long-press cancel action.

Controller mapping and three separate meanings

The node uses a fixed six-slot WebXR gamepad layout: trigger=0, grip=1, primary=4, and secondary=5. Buttons 2 and 3 are not mapped to operator events. The profile string is preserved for diagnostics but does not replace this implemented layout. The A/B/X/Y labels below follow the source comments and tests; the numeric slot is the wire-level fact.

Keep the following three controls conceptually separate. Global ARM is EVA collection's total gate. Right B is routed through the shared web:collect_arm:on/off commands; with ARM OFF, neither hand may make collection publish motion. Per-arm grip authorization is a hand-specific motion permission. Only a controller with grip_engaged=true makes that arm's spatial EEF target active; an unauthorized arm's joints are held at the previous safe value. Trigger gripper control is a gripper data channel. Its button-0 value is mapped to a gripper target and is not a synonym for either gate.

ARM OFF or a teleop reset clears both hands' permissions and accumulated pose references. Turning ARM back on does not restore old permissions. A new WebXR session also starts without the old controller state. This distinction is intentional: a gate transition must not silently re-arm motion, while a trigger may still describe a gripper target once a permitted application tick exists.

Relative retargeting, pose, and gripper behavior

Each WebXR pose contains a three-element position and an orientation_xyzw quaternion. Both node and client reject non-finite values, wrong dimensions, and a zero quaternion, and normalize valid quaternions. Accepted reference spaces are only local, local-floor, and bounded-floor. These are XR reference coordinates, not robot-world coordinates.

On the first update for an engaged arm, the retargeter latches the measured robot EEF as its home EEF and latches the controller's current position and rotation as the reference. That first output is home, so grabbing at an arbitrary headset pose does not jump the robot. Each later frame subtracts the previous controller position, composes the relative rotation, and accumulates both deltas. The position accumulation is transformed by base_from_xr_rotation, scaled by position_scale, and added to home. The orientation uses the accumulated relative rotation around the home orientation and is emitted in canonical qw qx qy qz order.

Releasing an arm's grip freezes its accumulated EEF target and clears that arm's current controller reference. Re-engaging at a new controller pose establishes a new reference and continues from the previous accumulated target, without a release/re-grab jump. A full reset clears home, reference position, reference rotation, accumulated position, accumulated rotation, and the filter. If tracking is lost while an arm is authorized, that tick is rejected; recovery continues from the existing anchor rather than re-anchoring to a bad pose.

eef_filter_alpha is the low-pass coefficient in (0,1]; 1.0 disables smoothing. Smaller values smooth position and quaternion jitter, while the gripper value stays responsive. An optional workspace supplies finite three-dimensional min and max bounds. An out-of-bounds target is rejected by the client and is not sent downstream.

Configuration fields and a shortened official example

VR fields live under collection.teleop. During validation, EVA compares client arm group names with the robot's runtime arm groups. The arms mapping must contain exactly those names, and every group must bind to a unique left or right controller. The parser rejects equal endpoints, non-positive timeouts, invalid rotation matrices, invalid workspace bounds, and gripper thresholds outside [0,1].

The following shortened block is taken from the official ARX X5 VR preset (its common fields also appear in the Dual Franka preset). It omits robot base configuration, ARX X5 cameras, and tasks. Use the target robot's official file when editing values: AgiBot G2's official gripper endpoints are open_value=0.0 and close_value=-0.785, so its endpoints must not be replaced casually with the 1.0/0.0 values shown here.

collection = dict(
    teleop=dict(
        _delete_=True,
        control_source="client",
        safety=dict(
            max_qpos_step=0.08,
            max_position_error_m=0.08,
            max_orientation_error_rad=0.35,
        ),
        client=dict(
            type="vr_webxr",
            endpoint="tcp://127.0.0.1:8765",
            ack_endpoint="tcp://127.0.0.1:8766",
            input_timeout_s=0.25,
            heartbeat_timeout_s=2.0,
            position_scale=1.0,
            eef_filter_alpha=0.1,
            base_from_xr_rotation=[
                [0.0, 0.0, -1.0],
                [-1.0, 0.0, 0.0],
                [0.0, 1.0, 0.0],
            ],
            gripper=dict(
                mode="linear", threshold=0.6,
                open_value=1.0, close_value=0.0,
            ),
            arms=dict(
                left_arm=dict(controller="left"),
                right_arm=dict(controller="right"),
            ),
        ),
    ),
)

ARM, FK/IK, and publish-time safety

When the console switches to COLLECT, EVA can prewarm the client teleop FK/IK path. It uses the robot's initial qpos for one FK→IK→FK validation and publishes no robot action during prewarm. Each live tick reads the latest transport qpos, computes measured EEF, polls the VR client, solves IK with the current qpos as seed, and validates the forward-kinematics residual before publishing.

The client exposes an explicit TeleopResult: only COMMAND enters IK and publishing, IDLE means no action is available, and REJECTED means that frame or target is unsafe. A stale, tracking-loss, workspace, or IK tick is dropped without automatically restarting the collection lifecycle. Read the status before proceeding or deactivating teleop.

Freshness, sessions, sequence, and fail-closed reconnects

Safety depends on message health as well as pose values. The node emits a heartbeat about every 0.25 s. EVA reports the source as connected only when the worker is healthy, the browser is connected, and the most recent node heartbeat is no older than heartbeat_timeout_s=2.0 seconds. A frame's seq must be a non-negative integer greater than the previous frame; an old or duplicate sequence is ignored. Every frame has a non-empty session_id, and an event must belong to the active session and is deduplicated by event id.

• Stale frame: with the official input_timeout_s=0.25, a frame older than 250 ms causes polling to report VR input frame stale ... and publish nothing for that tick. validate_result also rejects an expired source token.
• Heartbeat or browser disconnect: when the node reports browser_connected=false or heartbeat freshness expires, the client clears frame, session, authorization, and retarget anchors. A worker failure fails closed; old frames cannot continue motion.
• Session restart: a new session clears events, sequence, accumulated position, accumulated rotation, and per-arm engagement. If an earlier session existed, every bound controller must first provide a fresh neutral frame: valid, grip released, trigger no higher than min(gripper_threshold, 0.05), and no pressed controls. Only after that neutral frame may a later frame command motion.
• Publish race: if generation, connection, or source frame changes while IK is solving, validate_result fails and publish is skipped. An old solution cannot land on the robot after a reconnect.
• Event and ACK bounds: node event retry is scoped to the live browser session and defaults to a 2.0 s retry TTL. The event queue records a source error when full; node and client acknowledgement queues remain bounded (older feedback may be dropped), and a client event-buffer overflow fails closed. An ACK confirms handling; it never re-authorizes motion.

Troubleshoot in order

Check the layers in this order: device, page, node, ZMQ, EVA, then robot. Treat token, session, sequence, and source-error values in logs as current-session diagnostics; do not commit the token to a public script or repository.

Success criteria and collection output

A safe live-control check has four layers of evidence. The node remains ready and its VR RX seq increases. EVA reports connected=true, input age below 250 ms, and a healthy heartbeat. After ARM ON, every intended arm appears in authorized_groups and engaged_groups, and the published-action count increases. The first engaged frame starts at the measured EEF without a jump, and later targets stay inside the workspace and IK/FK limits.

For a collection take, short-release right A to start, perform the task, and short-release right A to stop; wait until COLLECT returns to idle. A successful episode adds data/chunk-000/episode_NNNNNN.parquet, one videos/chunk-000/<camera>/episode_NNNNNN.mp4 under each configured camera directory, and a line in meta/episodes.jsonl under the task dataset. A right-A long cancel should write nothing for that take.

Next steps

• Learn the collection schema, episode layout, and quality flags: Data collection.
• Edit inherited robot and EEF/IK settings: Configuration.
• Review COLLECT controls, status, and replay: COLLECT.
• For RL REAL/HIL intervention, complete setup and use the left-Y intervention control in the RL workspace.
• For physical Leader/Follower input, use the corresponding hardware path. Do not route a robot outside the three official VR presets through this client.