FreeTacMan
Robot-free Visuo-Tactile Data Collection System
for Contact-rich Manipulation
Good things take time... and so does loading this page! Thanks for your patience! Best viewed in Chrome on a desktop.
Enabling robots with contact-rich manipulation remains a pivotal challenge in robot learning, which is substantially hindered by the data collection gap, including its inefficiency and limited sensor setup.
Motivated by the dexterity and force feedback of human motion, we introduce FreeTacMan, a robot-free and human-centric visuo-tactile data collection system to acquire robot manipulation data accurately and efficiently. Our main contributions are:
1. An in-situ, robot-free, real-time tactile data-collection system that leverages a handheld gripper with modular visuo-tactile sensors to excel at diverse contact-rich tasks efficiently.
2. A large-scale, high-precision (sub-millimeter) visuo-tactile manipulation dataset with over 3000k visuo-tactile image pairs, more than 10k trajectories across 50 tasks.
3. Experimental validation shows that imitation policies trained with our visuo-tactile data achieve an average 50% higher success rate than vision-only approaches in a wide spectrum of contact-rich manipulation tasks.
Dive into our 💡interactive 3D model viewer and explore the most popular native 3D formats with ease.
Try out the 🖱️move command to inspect internal structures.
It's more than just viewing — it's a hands-on exploration. Start 💫discovering now!
FreeTacMan features a universal gripper interface with quick-swap mounts compatible with various robots, such as Piper and Franka, with support for more platforms coming soon. It also includes a camera scaffold designed for precise alignment with the wrist-mounted camera, ensuring a consistent perspective. These components demonstrate the plug-and-play modularity of FreeTacMan, enabling seamless integration across diverse robotic platforms without requiring hardware-specific modifications.
Enabled by the efficient, precise, and fidelity tactile data collection system, we curate a diverse dataset of manipulation tasks spanning vision, touch, and proprioception modalities. The dataset spans 50 tasks, comprising more than 10k manipulation trajectories which contains over 3 million visuo-tactile image pairs.
We evaluate the effectiveness of FreeTacMan system and the quality of the dataset through a diverse set of contact-rich manipulation tasks. We integrate tactile feedback to assess its impact on policy performance, observing a substantial improvement that highlights its dynamic value in contact-rich tasks.Temporal-aware pretraining further enhances performance by aligning visual and tactile embeddings while capturing temporal dynamics. Across five evaluated tasks, imitation policies trained with our visuo-tactile data achieve an average success rate that is 50% higher than vision-only counterparts.
The robot grasps a plastic cup and places it stably on a tray without causing damage.
The videos are played at normal speed.
Ours-α: + Tactile
Ours-β: + Tactile Pretrained
We evaluate the usability of FreeTacMan through a user study involving 12 human participants with varying levels of experience, each collecting demonstrations across 5 tasks. Compared to previous setups, FreeTacMan consistently achieves the highest completion rates and efficiency, and is perceived as the most user-friendly and reliable data collection system.
Fragile Cup
USB Plug
Texture Classification
Stamp Press
Calligraphy
P.S.: Completion per Unit Time (CPUT), defined as completion_rate x efficiency