Which Physical Tactile Cues Matter for Contact-Rich Imitation Learning?

A Controlled Real-Robot Study

Xuhao Qin, Xiyan Huang, Haoyu Dong, Xi Zhang, and Chenxi Xiao†

† Corresponding author, School of Information Science and Technology, ShanghaiTech University

PDF Video Code

Abstract

Vision-based tactile sensors capture multiple aspects of physical interaction within a single image stream, including contact appearance, geometry, indentation depth, and tangential deformation. However, existing visuo-tactile policies often directly process tactile observations as images, making it difficult to determine which physical cues contribute to policy learning. This paper presents a controlled real-robot study that compares physically interpretable tactile signals using the LeRobot platform, enhanced with our visuo-tactile perception system. From the same tactile stream, we derive tactile RGB images, normal/depth maps, and marker-flow fields, representing contact appearance, geometry, and tangential deformation, respectively. Using identical task demonstrations and the same policy backbone, we evaluate the individual and combined contributions of these tactile cues. We further compare tactile encoders and visuo-tactile fusion strategies. Evaluation across 720 real-robot trials yields four key findings: (1) tactile perception substantially improves the overall success rate from 57.8% to 78.9%, although its contribution varies across task stages; (2) marker flow is the most effective individual cue, achieving a success rate of 74.4% and playing a particularly important role in force regulation; (3) cue-factorized encoding consistently outperforms direct tactile-RGB encoding, highlighting that different physical cues require tailored processing techniques; and (4) cross-attention consistently improves performance across both encoders and all three tasks, offering a lightweight solution with performance comparable to that of pretrained tactile encoders.

Video

A two-minute overview of the LeRobot-compatible platform, tactile cue representations, controlled policy study, and real-robot evaluation.

Overview

Physical tactile cues for contact-rich imitation learning

LeRobot-compatible visuo-tactile platform, signal representations, controlled studies, and three real-robot tasks
This work investigates how tactile signals should be represented, encoded, and fused for contact-rich imitation learning. The same tactile observation is converted into tactile RGB, normal/depth, and marker-flow representations for controlled comparisons across three real-robot manipulation tasks.

LeRobot-compatible platform

A unified hardware and data pipeline synchronizes global vision, wrist vision, robot state, actions, and tactile observations.

Controlled cue study

The same demonstrations and policy backbone isolate the contributions of appearance, geometry, and tangential deformation.

Real-robot evaluation

Three tasks cover in-gripper adjustment, sustained surface contact, and tight-clearance insertion.

System and Policy

A shared Diffusion Policy pipeline keeps observations, action space, policy backbone, horizons, training data, and training budget fixed across the controlled studies.

Real-system platform and controlled Diffusion Policy pipeline
Overview of the LeRobot-compatible visuo-tactile system and Diffusion Policy pipeline. Tactile RGB and geometry use image encoders, marker flow uses an MLP, and wrist-camera and tactile-RGB features can optionally interact through cross-attention before multimodal fusion and action prediction.

Tactile Signal Representations

01

Tactile RGB

Raw tactile appearance preserves local contact patterns and color changes induced by elastomer deformation.

02

Normal / Depth

Surface normals and reconstructed depth describe contact geometry and local indentation.

03

Marker Flow

Tracked marker displacement exposes tangential deformation and established-contact regulation.

Exploded and assembled views of the tactile sensor
Sensor design. Exploded view of the internal components and the assembled CAD model.
Tactile RGB, depth, normal, and marker-flow examples for scissors, screws, and a USB flash drive
Physical tactile cues. From left to right, each row shows the contact object, tactile RGB image, depth map, surface-normal map, and marker-flow field.

Real-Robot Tasks

Tube reorientation and insertion

Adjust the pose of a tube inside the gripper and insert it into a holder while local contact changes remain partially hidden from external cameras.

Blackboard wiping

Maintain stable surface contact while moving tangentially across the target region, making pressure regulation and shear-related cues important.

Peg-in-hole insertion

Align and insert a peg under tight clearance, where small contact errors can cause jamming or failure.

Tube reorientation

In-gripper pose adjustment followed by insertion.

Blackboard wiping

Sustained contact with tangential motion.

Peg-in-hole insertion

Tight-clearance alignment and insertion.

Successful trajectories and cue-specific failure cases across three tasks
Representative successful trajectories and cue-specific failure cases across the three real-robot tasks illustrate how different tactile inputs are associated with failures at different interaction stages.

Cue-Specific Failure Cases

Representative failures reveal how each tactile representation supports different stages of contact-rich manipulation. Videos load only when played.

Tube reorientation and insertion

Tactile RGBJammed
GeometryWedged
Marker flowMisgrasped
Full tactileDropped

Blackboard wiping

Tactile RGBResidual
GeometryStalled
Marker flowSkewed
Full tactileResidual

Peg-in-hole insertion

Tactile RGBMissed
GeometryStalled
Marker flowMisgrasped
Full tactileMisinserted

Results

Controlled comparison of tactile representations

Experiment 1 · 450 real-robot trials. Five input configurations are evaluated on three tasks with 30 trials per configuration and task. Full tactile raises aggregate success from 57.8% to 78.9%, while marker flow is the strongest individual cue overall at 74.4%.

Input representationTube reorientationBlackboard wipingPeg insertion
Vision only14/30 (46.7%)18/30 (60.0%)20/30 (66.7%)
Vision + tactile RGB22/30 (73.3%)24/30 (80.0%)20/30 (66.7%)
Vision + normal/depth20/30 (66.7%)19/30 (63.3%)21/30 (70.0%)
Vision + marker flow22/30 (73.3%)23/30 (76.7%)22/30 (73.3%)
Vision + full tactile23/30 (76.7%)25/30 (83.3%)23/30 (76.7%)

Tactile encoder and fusion strategy

Experiment 2 · 270 additional real-robot trials. Three new encoder/fusion configurations are evaluated on three tasks with 30 trials per configuration and task. The ResNet · Concatenation row is the shared full-tactile baseline from Experiment 1 and is repeated below for comparison, so it is not counted again in the 720-trial total. With full tactile fixed, cross-attention improves aggregate success by 4.4-6.7 percentage points.

Encoder / fusionTube reorientationBlackboard wipingPeg insertion
ResNet · Concatenation23/30 (76.7%)25/30 (83.3%)23/30 (76.7%)
ResNet · Cross-attention24/30 (80.0%)27/30 (90.0%)26/30 (86.7%)
Sparsh-DINO · Concatenation26/30 (86.7%)25/30 (83.3%)24/30 (80.0%)
Sparsh-DINO · Cross-attention27/30 (90.0%)27/30 (90.0%)25/30 (83.3%)

Conclusion

This work presents a controlled real-robot comparison of physically interpretable tactile cues for contact-rich imitation learning. By deriving tactile RGB, normal/depth geometry, and marker-flow fields from the same sensor stream and evaluating them with matched demonstrations and policy settings, the study separates the contribution of physical cues from the effects of tactile encoding and visuo-tactile fusion.

  1. Tactile feedback provides a substantial overall benefit.

    Across 720 real-robot trials, full tactile perception achieves 71/90 successes (78.9%), compared with 52/90 (57.8%) for vision only, an aggregate improvement of 21.1 percentage points. The benefit is nevertheless task- and stage-dependent.

  2. Marker flow is the strongest individual tactile cue.

    Marker flow reaches 67/90 successes (74.4%) and is particularly informative for regulating force after contact has been established. The qualitative failures also show that different cues support different stages of contact establishment and regulation.

  3. Full tactile adds modest final-success gains but improves execution continuity.

    Full tactile exceeds marker flow by only 4.4 percentage points overall and by one successful trial in each task. Its primary benefit is smoother, more continuous transitions between contact establishment, regulation, and task completion.

  4. Cue-factorized encoding outperforms direct tactile-RGB encoding.

    Instead of processing the tactile observation as a single RGB image, cue-factorized encoding separately processes tactile appearance, normal/depth geometry, and marker flow. This tailored treatment of different physical cues achieves better performance than direct tactile-RGB encoding.

  5. Cross-attention provides a robust and lightweight fusion improvement.

    With the full-tactile input fixed, cross-attention consistently improves success across all three tasks for both encoders, raising aggregate success by 6.7 points with ResNet-18 and 4.4 points with Sparsh-DINO. Its gains are stable and its performance is comparable to tactile-pretrained encoding; encoder choice, by contrast, has task-dependent effects.

Limitations and future work. The present study is limited to one vision-based tactile sensor and a parallel-jaw gripper platform. Future work will examine whether these cue-dependent findings generalize across tactile sensor designs, robot embodiments, and broader contact regimes.

Read the paper

BibTeX

@misc{qin2026physical,
  title        = {Which Physical Tactile Cues Matter for Contact-Rich Imitation Learning? A Controlled Real-Robot Study},
  author       = {Qin, Xuhao and Huang, Xiyan and Dong, Haoyu and Zhang, Xi and Xiao, Chenxi},
  year         = {2026},
  howpublished = {Project website},
  url          = {https://xuhaoqin02.github.io/lerobot-tactile/}
}

This preliminary citation can be updated with the final venue, DOI, or arXiv identifier when available.