P2P-T

Learning tool use from human demonstrations

From Pixel
to Poses. Object-centric tool manipulation learning from human demonstrations

P2P-T learns embodiment-independent 6D tool pose priors from in-the-wild human video, then turns those priors into precise robot actions without human-robot aligned data.

Demos arXiv GitHub Contact
Bangjun Wang1 Longyan Wu2 Yukun Wei1 Shenghe Shao1 Chaoyi Huang1 Wenze Cui1 Zetong Xu1 Hanlin Wu1 Long Chen3 Yi Ma1 Hongyang Li1,2
1 The University of Hong Kong 2 Shanghai Innovation Institute 3 Xiaomi EV
arXiv preprint · September 2026

01At a glance

Teach robots tool use through object motion

P2P-T turns unstructured human videos into structured, object-centric pose trajectories. A foundation-model pipeline finds the tool, reconstructs its geometry, and tracks its 6D motion. A two-stage learner then transfers those priors to a low-level robot policy, keeping the task affordance while leaving embodiment-specific control to the robot.

2 stagesworld model → policy
3,094 clipspretraining corpus
17 toolscategories in the mix
0.57average success rate
P2P-T overview: human video preprocessing, object-centric world model pretraining, and pose-aware deployment
From in-the-wild human demonstrations to pose-aware robot execution.
01

Automated pose extraction

Grounded-SAM, SAM 3D, and FoundationPose convert raw RGB video into clean tool masks, meshes, and 6D trajectories.

02

Embodiment-independent priors

The Stage 1 world model predicts future object poses, so the learned dynamics transfer across human and robot bodies.

03

Pose-aware execution

Stage 2 injects future pose latents into a diffusion action expert for precise, receding-horizon control.

02Method

Two stages bridge pixels and physical action

Stage 1

Object-centric world model

Frozen DINOv2, SigLIP, and T5 encoders feed an autoregressive transformer that predicts future 6D tool poses and auxiliary frames from human demonstrations.

Stage 2

Pose-aware post-training

A frozen Stage 1 model produces future pose latents. An RDT-based action expert conditions on those latents, language, images, and proprioception.

Deployment

Receding-horizon control

The robot samples action chunks and refreshes its pose prior as new camera observations arrive, connecting object-level dynamics with robot-specific control.

P2P-T two-stage framework diagram
Two-stage framework: object-centric world model pretraining followed by pose-aware policy post-training.
Whole architecture

P2P-T separates embodiment-independent reasoning from robot-specific control. Stage 1 learns future object-centric pose latents from human demonstrations; Stage 2 conditions a low-level action expert on those latents together with current RGB, language, and proprioception.

P2P-T preprocessing pipeline from video frames to 6D tool poses
Automated preprocessing removes the need for dense human annotation.
Preprocess pipeline

The pipeline turns unstructured RGB video into validated 6D tool trajectories: Grounded-SAM isolates the tool, SAM 3D reconstructs an instance mesh, and FoundationPose tracks its SE(3) pose across the sequence. The resulting supervision avoids dense human-robot alignment.

03Robot demonstrations

Precision tool use beyond pick-and-place

P2P-T real-world evaluation tasks: hammer, cup, brush, screwdriver, knife, and spoon
Six task demonstrations spanning impact, pouring, sweeping, slicing, scooping, and tool alignment.

Six real-world tasks

Each task requires continuous spatial reasoning, contact-rich control, and a tool-specific working pose. Tools are randomly placed, then re-oriented and operated by a UR10e robot with a Robotiq 2F-140 gripper.

Hammer
Strike a target with dynamic impacts.
Cup
Pour while maintaining orientation through the motion.
Brush
Sweep across a surface with sustained contact.
Screwdriver · Knife · Spoon
Rotate, slice, and scoop through precise 6D trajectories.

Real-world execution

P2P-T's real-world inference results on UR10e.

Hammer

Strike a target with dynamic impacts.

Cup

Pour while maintaining orientation through the motion.

Knife

Slice with a precise downward working direction.

Brush

Sweep across a surface with sustained contact.

Spoon

Scoop while coordinating pitch and tool orientation.

Screwdriver

Rotate and align the tool with the screw.

04Evaluation protocol

Trajectory inference from random tool poses

For each task, we randomly place the tool on the table, requiring the robot to adapt to its initial position and pose, infer the corresponding trajectory, and progressively reorient the tool into the correct working pose while maintaining contact or precise directional control. This evaluates tool use from realistic, unaligned starting conditions rather than simplified, pre-aligned setups. We use 100 demonstrations for adaptation and evaluate each task over 50 trials.

Random placement - inferred trajectory
Random placement - inferred trajectory
Random placement - inferred trajectory

05Results

Strong gains on complex real-world manipulation

0.57average success
+73%over π0.5
30 Hzexecution rate

Real-world policy evaluation

Success rate (%) · 100 demonstrations and 50 trials per task. Selected baselines shown.

BaselinesP2P-T

Hammer

24%π0.521%Octo59%P2P-T

Cup

49%π0.538%Lingbot-VA60%P2P-T

Brush

27%π0.527%Octo55%P2P-T

Screwdriver

35%π0.536%Lingbot-VA54%P2P-T

Knife

37%π0.524%OpenVLA57%P2P-T

Spoon

24%π0.526%Octo58%P2P-T
Scaling verification
VariantParametersHammerCupAverage
P2P-T-Small1.189B0.340.480.41
P2P-T-Base1.921B0.590.600.60
P2P-T-Large2.832B0.640.680.66
Ablation on framework design
Object-centric pretrainingPose-aware post-trainingHammerCupAverage
✓×0.120.140.13
×✓0.320.360.34
✓✓0.590.600.60

Both components are needed for the strongest pose-aware execution performance.

06Qualitative analysis

Structured priors carry tool affordances across bodies

From affordance to action

Human demonstrations provide the object-level trajectory: where a tool should move, how it should orient, and how its contact evolves. P2P-T keeps that structure while learning robot-specific actuator commands in Stage 2.

Failure cases

Why pose priors matter

In most cases, when tools are placed in initial poses that were not seen during training, baseline models linger around the target and cannot complete a reliable grasp. In other cases, they overfit to a demonstrated trajectory, grasp a random component of the tool, and fail to use the tool correctly. P2P-T's future pose latents make the intended orientation and progression explicit, helping the action expert maintain a stable working trajectory.

These clips show representative failures from the baseline comparison. The pose-aware prior keeps the tool aligned with its intended affordance.

ScrewdriverThe policy fails to generalize to a new initial tool pose and uses the screwdriver incorrectly.
HammerThe interaction misses the target working pose.
KnifeThe end-effector loses the intended cutting direction.

07Citation

BibTeX

@article{wang2026p2pt,
  title={From Pixel to Poses: Object-centric Tool Manipulation Learning from Human Demonstrations},
  author={Wang, Bangjun and Wu, Longyan and Wei, Yukun and Shao, Shenghe and Huang, Chaoyi and Cui, Wenze and Xu, Zetong and Wu, Hanlin and Chen, Long and Ma, Yi and Li, Hongyang},
  journal={arXiv preprint},
  year={2026}
}