← All model news

DexRoam trains robots from first-person human demos: report cites 29% to 56% average success, robot demos halved

According to a November 5, 2026 report by Machine Heart, HKUST, the Beijing Academy of Artificial Intelligence, Peking University and others proposed DexRoam, which trains mobile bimanual dexterous manipulation from first-person human whole-body demonstrations. After three-stage alignment, "human pretraining plus human-robot joint fine-tuning" raised GR00T N1.7 average success from 29% to 56% and π₀.₅ from 32% to 57%; halving robot demonstrations stayed close to the full baseline. Code for collection, alignment and policy training is reported open-sourced.

Published Updated

What changed

According to a November 5, 2026 report by Machine Heart, a joint team from Hong Kong University of Science and Technology, the Beijing Academy of Artificial Intelligence, Peking University and other institutions proposed DexRoam, a system covering first-person human demonstration capture, whole-body motion alignment, robot policy learning and real-robot execution; the report says the work was accepted to CoRL 2026. On the capture side, the report says the system needs no external motion-capture rig: a demonstrator wearing a Meta Quest 3 headset and a head-mounted ZED Mini stereo camera can synchronously record first-person video, whole-body joint motion and hand keypoints, with the camera feed echoed live to the headset to keep visual observations consistent with policy training; the team also built a single-operator whole-body teleoperation system using a VR headset, controllers and a data glove to synchronously control the robot's base, torso, arms, head and dexterous hands. Alignment has three steps: body alignment, using the human pelvis as reference to build a coordinate frame and retarget the hands; action semantic alignment, rewriting base, torso and arm motion as relative action representations; and temporal alignment, resampling human trajectories by task progress to match the robot's time scale. The report says that after alignment, the MMD and SWD metrics of human versus robot action distributions fell by an average of 41.7% and 31.1% respectively, and that converted human trajectories could be successfully replayed on the real robot. On the policy side, the team used two VLA models, GR00T N1.7 and π₀.₅, to compare three training schemes: human-robot joint training, human pretraining followed by fine-tuning on robot data, and human pretraining followed by human-robot joint fine-tuning. The report says all three improved results across five tasks, with the third best: GR00T N1.7 average success rose from 29% to 56% and π₀.₅ from 32% to 57%. The team also says that with the same number of human demonstrations and half the robot demonstrations, task performance was close to the baseline using full robot data, phrasing this potential as replacing roughly 50% of teleoperation data; one behavior seen only in human demonstrations was switching to a side grasp on a chip can depending on its orientation, whereas policies trained only on robot data favored a frontal grasp. The report says the capture and alignment code and the policy training code are both open-sourced, and gives paper and project addresses.

What it means for developers

For developers doing embodied model fine-tuning and data pipelines, the point of this news is not a new model but the possibility that the source of training data may change structurally. The report's core claim is that, after three-stage alignment, human whole-body demonstrations can share the same observation-action interface as robot demonstrations in VLA policy training, without extra model-structure changes for human data; so in mobile bimanual settings where teleoperated demonstration collection is costly or even needs multiple operators, first-person human demonstrations are framed as a scalable supplementary data source that may replace part of expensive robot teleoperation data. Keep the levels distinct: human data comes from a self-built portable device combination, robot data from a self-built teleoperation system, and GR00T N1.7 and π₀.₅ are existing third-party VLA models. DexRoam delivers a method and code, not a hosted service you can call directly or a new set of weights. Its effect on integration is therefore indirect: if your post-training or evaluation pipeline lacks whole-body coordination demonstrations, it offers an alternative path at the capture and alignment stage; if you only call some VLA through an API for inference, the reported conclusions do not change your request parameters, concurrency behavior or billing structure. What really needs reassessment is data budget allocation: when human demonstrations can cover the task semantics and the aligned motions are executable on your embodiment, shifting part of the collection budget from teleoperation to human demonstration is a reasonable candidate; conversely, if the task depends on contact forces or precision assembly that are hard to reproduce in human demonstrations, this substitution has no basis.

Integration and evaluation

The following three points are conditional suggestions and inferences based on the above report, not conclusions in the report, and are not independently verified; treat them as hypotheses to test, not conclusions. First, set a replay gate on the alignment stage before deciding whether human data enters training. The corresponding reported change is that three-stage alignment cut MMD and SWD by an average of 41.7% and 31.1%, and converted human trajectories could be replayed on the real robot. The choice to make is whether to bring human demonstrations in as an upstream data source on your own embodiment, rather than tuning the training recipe first. The precondition is that you can obtain the joint mapping and hand retargeting rules for the target embodiment and can record first-person whole-body data. A measurable criterion: on your task set, compare the MMD/SWD of human versus robot action distributions before and after alignment, and measure the real-robot replay success rate of human trajectories (for example, a fixed sample of 20 trajectories; if the replay success proportion is below the majority, the mapping has not converged), and if replay falls short, do not proceed to policy training. Second, try "human pretraining plus human-robot joint fine-tuning" first, but confirm with a controlled comparison rather than watching demo videos. The corresponding reported change is that this combination was best among the three schemes, raising the two models' average success from 29% to 56% and from 32% to 57%. The choice is which of the three recipes to run first. The condition is your volume of robot demonstrations: prioritize this combination when robot data is scarce; if robot data is already ample and the target task differs greatly from human demonstration motions, direct human-robot joint training may be the easier starting point. A measurable criterion: fix the number of robot demonstrations (for example, compare across schemes at equal counts), compare average success and task completion of the three recipes on the same evaluation tasks, and record variance rather than a single best run. Third, use a data-halving curve to decide whether the next collection budget goes to human demonstrations or teleoperation. The corresponding reported change is that with robot demonstrations halved, task performance stayed close to the full baseline. The choice is which side to fund. The precondition is that the target task can be covered by human demonstrations; if it involves high-precision force control or operations unsuitable for humans to demonstrate, that precondition fails. A measurable criterion: step robot demonstrations down to 100%, 50% and 25%, plot the success-rate curve, and only when the 50% level still falls within your acceptable performance range should you shift budget to human collection; otherwise keep investing in teleoperation.

Limits and open questions

The reported success rates, MMD/SWD reductions and robot-data-halving conclusions all come from the team's own five-task experiments, with no independent reproduction seen; the report does not detail the composition of the five tasks, the robot platform model used or the full evaluation protocol, so cross-embodiment transfer needs re-verification. Human alignment quality depends on calibration of the capture devices and demonstrators' habits, and the report gives no stability data across demonstrators. The open-source scope needs to be distinguished: the report only says the capture and alignment code and the policy training code are open-sourced; it does not state specific license terms, nor a robot hardware list or purchasable configuration; open-sourced code is not the same as open-sourcing any model weights. GR00T N1.7 and π₀.₅ are existing third-party VLA models, and whether their weights are open and under what conditions is decided by their respective publishers and cannot be inferred from this report. Availability note: as of the catalog check on November 12, 2026, DexRoam has no corresponding entry in the KimiSeek catalog, and GR00T N1.7 and π₀.₅ likewise have no exact-match entries; the report also mentions no vendor-hosted API, so the way to obtain it is via the team's public repositories and paper. If related vendor interfaces or hosted services appear later, their availability scope and pricing should be governed by that publisher's statements, not by this catalog.

Three minutes to your first working call

Create an account and an API key. Calls use your account balance. 30% off your first top-up. Top up ¥500, pay ¥350, receive ¥500 in credit. Top up ¥1000, pay ¥850, receive ¥1000 in credit You can still start with a ¥10 trial top-up.

Sources and editorial process

CoRL 2026|移动双臂灵巧操作,也能从人类示范中学?

AI assisted the analysis and translation of the cited sources, followed by automated quality checks. Recommendations are not test results. Consult the current catalog for availability and pricing.