Dataset formats and quality export

Separate recording from export

EVA has two deliberately separate stages. Between START and STOP in COLLECT, the capture path receives teleoperation data. A save worker aligns, validates, and packs the captured snapshots, then writes a LeRobot v2.1 episode. The recording path does not branch on the format selector and does not write v3.0, HDF5, or MCAP in real time. Those formats are produced only after a v2.1 dataset already exists, when the quality split and the selected offline exporter run. This boundary keeps capture timing, replay behavior, and raw evidence stable, while allowing the same reviewed source to be exported for different consumers.

The conversion registry contains four format values: native lerobot_v21, lerobot_v3, hdf5, and mcap. The last three are the new export focus here. Selecting native v2.1 still runs the quality split and creates accepted/rejected copies; it is not a shortcut that treats an unreviewed raw directory as a published dataset.

Complete the lifecycle in COLLECT

Launch EVA with a config that contains a collection section. The COLLECT panel presents the dataset name and task prompt separately. Its DATASET NAME and TASK / PROMPT values come from collection.tasks. The selected dataset is stored at <log_dir>/<dataset-name>/raw/; when log_dir is empty, the configuration loader derives a work-directory path before the dataset name and raw suffix are applied.

After START, capture, saving, and image encoding use background work so live control is not blocked. If the host falls behind, the queue grows and a warning is logged. STOP validates, packs, and queues the episode; a zero-frame episode writes nothing. Relaunching into the same collection dataset appends later episodes rather than overwriting earlier raw episodes. CANCEL clears the in-flight snapshots; it is not a failed export and creates neither an accepted nor a rejected directory.

Split accepted and rejected episodes

Each line in the raw v2.1 meta/episodes.jsonl describes one episode. During collection saving, an episode with no automatic issue gets quality: "green"; any automatic issue produces quality: "red" and a quality_issues list. The QUALITY CHECK action writes an operator decision as qc_verdict: "pass" or qc_verdict: "fail" and can replace the note. A note-only update may have no verdict. At export time the source implementation applies this priority:

Accepted and rejected are complete dataset directories. Each contains the subset's renumbered episodes, its video or embedded-image representation, task data, statistics, source metadata, and the quality marker. The split does not erase the original quality reasons. Within each subset, episode indices are renumbered from zero; Parquet episode_index and global index values are rewritten to contiguous values so a consumer can read the subset directly.

Read quality_split.json

Every output directory contains meta/quality_split.json. This is provenance and upload-gating metadata, not an optional decoration. Its four important fields are:

For example, if accepted contains [0, 3], accepted episode 0 and accepted episode 1 came from raw episodes 0 and 3. They are not aliases for raw episodes 0 and 1. This list preserves traceability even though each subset starts its own numbering at zero.

Stage, publish as a pair, and roll back

Quality export does not expose half-written final directories. The exporter creates temporary staging roots for accepted and rejected. For non-native formats it first makes a temporary native v2.1 split, then writes both target-format subsets into separate staging directories. Only after both subsets and their metadata have been written does the publisher move them to their final paths. The paths are derived from the current source directory and format.

<log_dir>/<dataset-name>/raw/
└── ...                                  # v2.1 data from live collection

<log_dir>/<dataset-name>/export/<format>/
├── accepted/                            # current, publishable subset
└── rejected/                            # current, rejected subset

The tree above applies when the source directory is named exactly raw. If its name is not raw, the server uses <source-name>_export/<format>/accepted and the matching rejected path. Do not hard-code a different fixed path; the Console computes the paths and returns the actual directories in the export job.

Accepted and rejected are one logical publication pair. If validation, conversion, or publication of the second directory fails, temporary roots are removed and the final pair is not left half-published. When replacing an existing export, the publisher first moves the old directories to backups. If the later move fails, it removes any newly published directory and restores the old directories. A rerun therefore cannot leave a new accepted output beside an unrelated old rejected output. “Atomic publication” here means pair-level final-directory consistency; it does not mean an external reader can inspect a file while that file is being written.

Native LeRobot v2.1 raw layout

Before comparing the exporters, identify the evidence shape of the source. A selected live collection dataset is recognized by meta/info.json with codebase_version: "v2.1"; the source reader rejects a quality-split input whose codebase_version is not v2.1 or whose explicit dataset_format is not lerobot_v21 (an omitted format field is treated as v2.1).

<log_dir>/<dataset-name>/raw/
├── data/
│   └── chunk-000/
│       ├── episode_000000.parquet       # one row per frame
│       └── episode_000001.parquet
├── videos/
│   └── chunk-000/
│       └── <video-key>/
│           └── episode_000000.mp4       # one MP4 per camera per episode
└── meta/
    ├── info.json                         # v2.1, fps, columns, video features
    ├── episodes.jsonl                    # episode, task, length, quality, QC
    ├── episodes_stats.jsonl              # per-episode statistics when available
    ├── tasks.jsonl                       # task_index and task text
    └── stats.json                        # dataset statistics

Raw Parquet timestamp values use the configured target fps to form an even grid; capture_time keeps the observed capture time for diagnostics. Camera videos are intended to correspond one frame per table row. Raw saving keeps the collected vectors, images, and quality information; export does not infer a new robot action or silently replace a quality decision.

LeRobot v3.0: sharded tables and videos

When the selector is LeRobot v3, the exporter reads each accepted or rejected v2.1 episode, writes tables into v3-style Parquet files, and writes camera frames into MP4 files associated with the shards. This remains an offline operation. It leaves the raw v2.1 directory unchanged and does not produce v3 files during capture.

<export-root>/lerobot_v3/accepted/
├── data/
│   └── chunk-000/
│       ├── file-000.parquet             # a shard can contain several episodes
│       └── file-001.parquet              # rollover at a boundary or schema change
├── videos/
│   └── <video-key>/
│       └── chunk-000/
│           └── file-000.mp4              # that camera's frames for the shard
└── meta/
    ├── episodes/
    │   └── chunk-000/file-000.parquet    # episode → data/video files and ranges
    ├── tasks.parquet                     # Parquet form of the task table
    ├── info.json                          # dataset_format=v3, paths, features
    ├── episodes.jsonl, tasks.jsonl       # retained common metadata
    ├── stats.json
    └── quality_split.json

The data writer writes one episode at a time. It closes the current Parquet and video shard when adding the episode would push the Arrow table estimate over about 100 MiB, or when the schema changes, and then starts the next file-NNN. meta/info.json records the data and video file-size conventions plus data_path and video_path templates. Each episode metadata row records data/chunk_index, data/file_index, dataset_from_index, and dataset_to_index. When videos exist, it also records the video chunk and file indices and each video's from_timestamp and to_timestamp.

Do not read v3 with the v2.1 assumption “one Parquet and one MP4 per episode.” Read meta/episodes/chunk-000/file-000.parquet first, then follow its file-index and range columns to the relevant Parquet and camera MP4 shards. Tasks are available in both the copied tasks.jsonl and the writer-created tasks.parquet. A video whose decoded frame count differs from its episode table row count aborts the writer, so the accepted/rejected pair is not published.

HDF5: one file per episode

HDF5 puts an episode's non-image columns, image arrays, and episode metadata into one file. It does not retain independent MP4 files. The source v2.1 videos are decoded to frames and written as appendable HDF5 datasets in the images group. The exported meta/info.json changes those features to dtype: "image", sets embedded_images: true and total_videos: 0, and removes video_path.

<export-root>/hdf5/accepted/
├── data/
│   └── chunk-000/
│       ├── episode_000000.hdf5
│       └── episode_000001.hdf5
└── meta/
    ├── info.json, episodes.jsonl, tasks.jsonl, stats.json
    └── quality_split.json

episode_000000.hdf5
├── columns/                               # state, action, and other non-images
│   └── item_000000/{value, attrs[key,kind]}
├── images/                                # one appendable value per image key
│   └── item_000000/{value, attrs[key,kind]}
└── metadata/                              # episode row plus fps
    └── item_000000/{value, attrs[key,kind]}

The three top-level groups represent mappings whose keys are stored as a key attribute on each item_NNNNNN group. Numeric values generally use kind: "array". Strings, bytes, None, and values that cannot be represented as a direct numeric array carry a corresponding kind; mappings, lists, and tuples can be kept as MessagePack bytes so their metadata structure is not discarded. The metadata group also contains the episode's source metadata and source fps.

Each image value uses axis 0 as its frame axis and is appendable. The first frame establishes the remaining shape. If later frames for one image key change H×W×C shape, or the final observed frame count differs from the Parquet row count, the HDF5 writer raises an error. The file remains inside staging and is not published as a final accepted or rejected directory.

MCAP: EVA messages and embedded images

MCAP export creates one .mcap file per episode. It is intended for consumers that already read MCAP and want a message- and timestamp-oriented stream. The current implementation is not a ROS bag and does not use ROS message schemas. Do not interpret its topic names as ROS topic types or its payload as a ROS message.

<export-root>/mcap/accepted/
├── data/
│   └── chunk-000/
│       ├── episode_000000.mcap
│       └── episode_000001.mcap
└── meta/
    ├── info.json, episodes.jsonl, tasks.jsonl, stats.json
    └── quality_split.json

MCAP records
├── topic: episode                         # one episode metadata message
│   └── MessagePack({columns, images, metadata})
└── topic: episode/image/<video-key>       # one MessagePack image per frame
    └── sequence=frame_index; shared timestamp per frame

The writer uses MCAP profile eva-client and library eva-client. It registers eva_client.episode as a descriptor and then writes a channel named episode with message_encoding: "messagepack". That metadata message contains the non-image columns, image columns already inline in the source table, and {...episode row, fps} metadata. The registered descriptor is MCAP metadata from this implementation, not a ROS schema; the actual payload encoding is MessagePack.

Source videos are not copied as separate MP4s into the MCAP directory. Each camera receives an episode/image/<key> channel and each frame is encoded directly with MessagePack. For frame index 0, 1, 2, and so on, every camera uses the same timestamp computed from the target fps (the first timestamp is one frame period) and uses that index as the sequence number. Multi-camera messages can therefore be aligned by frame index and shared timestamp. The writer checks that no camera ends early and that no camera has trailing frames; a mismatch fails the export and prevents the pair from being published. The common metadata marks embedded images, sets total_videos: 0, and removes video_path.

Compare the four registered formats

The table describes structures the source actually writes. It does not call any format a universal training standard or claim that one format is generally superior. Native v2.1 is the live recording boundary; the other three are different consumer representations of the same reviewed source.

All exports retain common meta/info.json, task data, and statistics when the source files exist, and all write quality_split.json. For HDF5 and MCAP, do not look for a video directory and assume conversion failed; inspect embedded_images, total_videos, and the absence of video_path. For v3, use the episode locator metadata to interpret Parquet and MP4 shards instead of applying the v2.1 one-file-per-episode path.

Shared validation and verification

The exporter first validates the source as v2.1: meta/episodes.jsonl and meta/info.json must exist, their version and format must match, and episode indices must be unique. Every source Parquet and listed camera file must exist. A source table must contain episode_index and global index; every task index must resolve through tasks.jsonl; and each statistics column must be available from per-episode statistics or the table itself. Empty episodes are rejected by the format writers.

To verify an export, open meta/quality_split.json in both accepted and rejected first. Confirm that source_dir points to the current raw directory, subset is correct, dataset_format matches the selector, and source_episode_indices are the expected original indices. Compare info.json episode/frame totals with the physical output: for v2.1 compare each Parquet row count with each MP4 frame count; for v3 follow the episode locator's shard and range columns; for HDF5 inspect axis 0 of every images/*/value; for MCAP count image-topic messages and compare sequence and shared timestamps.

Then perform a real read test with the consumer's Parquet, HDF5, or MCAP reader: load one state value, one action value, and one image frame. Do not infer success from file extensions alone. An MCAP reader should see one episode metadata message and the camera channels episode/image/<key>; payloads are MessagePack, not a ROS bag/schema. After the read test passes, proceed through the upload gate.

Why upload is limited to the current accepted export

The UPLOAD ACCEPTED button is gated by the selected format and a completed export job. When the operator changes the format selector, the frontend clears its saved acceptedDir, export state, and upload state, then reports that an export is required. A completed HDF5 accepted directory therefore cannot be submitted as MCAP. Different formats have different directory layouts, image carriers, and info.json semantics; re-exporting is necessary to produce a matching artifact, not a redundant copy.

The server also checks the upload request. There must be a latest completed export job for the current source directory and requested format, and its accepted path must be the path being uploaded. The accepted directory's meta/quality_split.json must be readable; subset must be accepted; dataset_format must equal the request; and source_dir must resolve to the current source. A rejected directory, old format, mismatched source, or never-exported directory is refused. No upload thread starts when no upload backend is configured.

The currently resolved public upload backend is collection.storage.sftp. It needs a host, port, optional user and identity file, and an absolute canonical remote directory. The Console starts the job asynchronously and reports file and byte progress. SFTP first writes every file into a temporary staging directory under the same remote parent, then publishes by remote move. If the preferred target already exists, it does not replace that old target; it chooses a timestamped copy name. If upload fails, the new remote staging directory is cleaned and the new target is not published; an existing older target is left intact.

Handle failures

The panel reports queued, running, completed, or failed, with episode, file, and byte progress. A failed job's error text helps locate the issue; temporary files are not a success signal, and only final visible directories count as published output. If an operator uses PASS to override an automatic red flag, retain that decision and its note so a later reader can understand why the episode entered accepted.

Choose by the consumer's read path

Select based on the directory and reader that the downstream system actually needs, not on whether a format name sounds “standard.” One raw dataset can be exported more than once for different consumers. Each export must be verified independently, and only that export's accepted directory should be uploaded.

• Need EVA replay, raw collection evidence, or more review: keep the lerobot_v21 raw directory. It preserves one Parquet and independent camera MP4 files per episode plus collection quality fields and remains the source for later exports.
• Need sharded Parquet and sharded MP4: choose lerobot_v3, and make the reader follow the episode metadata's file indices, global index range, and timestamp range.
• Need episode-file boundaries with direct columns, images, and metadata: choose hdf5. Read the three groups and use axis 0 of the image datasets for frames; do not search for MP4 files.
• Need a message log or event-stream consumer: choose mcap. Consume the episode metadata message and camera image topics as MessagePack, using frame index and shared timestamp for alignment.

The source does not promise a universal training standard, an optimal format, or that newer means better. If a consumer supports several formats, compare its implemented decoders, concurrent-read behavior, image-memory needs, and metadata requirements. If it supports one format, follow that reader's contract. Keep the raw directory and both quality-subset markers alongside any published product so future users can trace the decision.

Delivery checklist

1. Confirm live recording has returned to IDLE and the raw directory contains v2.1 info.json, episode tables, camera files, and quality metadata.
2. Replay and resolve episodes in QUALITY CHECK, understanding the priority of green, red, PASS, and FAIL.
3. Select one format and press EXPORT DATASET. Wait for a completed job and record the accepted/rejected episode and frame totals.
4. Inspect both quality_split.json files for source, subset, format, and source episode indices. Inspect the target format's image carrier and directory layout.
5. Run the real downstream reader for a minimal read. For v3 check shard locator metadata; for HDF5 check all three groups and image axis 0; for MCAP check MessagePack topics, sequence, and timestamps.
6. Press UPLOAD ACCEPTED only when the selector has not changed, the current accepted marker matches, the SFTP backend is configured, and the export job is completed.