The Project will be executed using The Touch by Phantom Touch.

Conceptual cover page created with the assistance of ChatGPT.

Haptic Feedback for Fine-Tuning Vision-Language-Action Models

Ever wondered if you could “feel” your way through solving a robotic task like peg-in-hole insertion instead of just watching it? This project explores how adding haptic force feedback to a teleoperation system can change the way humans teach robots complex, contact-rich skills. We build a haptic interface using a modified 3D Systems Touch device connected to a MuJoCo simulation, allowing users to physically feel contact forces while controlling a virtual gripper. The goal is to understand whether this tactile feedback improves the quality of human demonstration data for imitation learning and whether it helps vision-language-action models (SmolVLA) learn faster and perform better in precision manipulation tasks. By comparing performance with and without haptics—and against a pretrained model baseline—we evaluate task success, speed, and contact accuracy in a peg-in-hole setup. We also prototype reversible stylus attachments to improve immersion and analyze system stability through impedance control theory. Ultimately, the project aims to show whether “feeling the robot’s world” can close the gap between human intuition and machine learning in fine motor control.

Introduction

Explain the motivation for your project in terms of the educational objectives and why your haptic device is an appropriate approach.

Background

Explain the relevant prior work in the field of haptics and provide references. These will likely be different from the references in your project proposal. Make sure to do a thorough literature search on relevant haptic devices/application.

Methods

Provide a detailed description of your project, such that another student from the class could generally re-create your project/experiment from the report if necessary. (You don't need to document every screw, but the design should be clear.) Add images and videos as needed to support the description. You can refer to downloadable drawings and code in the "Files" section (later). You should divide this section into subsections, which can vary depending on your particular project. Here is an example set of subsections:

Hardware Design and Implementation

System Analysis and Control

Demonstration / Application

Results

Describe the results, which may include qualitative responses from users at the open house.

Future Work

Describe how your system could be tested (e.g., through experiments if you have not already done so), how it can be improved, and how it might be applied.

Acknowledgments

Here you can list any individuals or groups who helped you with your project. (e.g., another student in the class, a course assistant, or an especially helpful PRL TA). Optional, so delete this section if you aren't using it.

Files

Code and drawings should be linked here. You should be able to upload these using the Attach command. If you aren't willing to share these data on a public site, please discuss with the instructor. Also, in this section include a link to a file with a list of major components and their approximate costs.

References

List the referenced literature, websites, etc. here.

Appendix: Project Checkpoints

1. Checkpoint 1 (May 14): Control and system dynamics drafted with relevant equations. Sketches of physical attachment ideas.
2. Checkpoint 2 (May 18): Finalized feasible top pick for physical attachment. Virtual environment set up in MuJoCo. Peg-in-hole environment implemented with correct collision geometry.
3. Checkpoint 3 (May 22): Touch device stylus position maps correctly to virtual gripper. Basic spring-wall force rendering is functional and stable. First physical attachment prototype fabricated and fit-tested on Touch device.
4. Checkpoint 4 (May 25): Impedance controller fully implemented (based on final physical attachment). Stability sweep complete (rendered stiffness varied across range, instability onset identified and compared to theoretical prediction). User study protocol finalized.
5. Checkpoint 5 (May 26): Data collection completed for both haptic and non-haptic human conditions (minimum 30 trials total). [BONUS] Pretrained SmolVLA running in MuJoCo evaluation loop with camera rendering functional.
6. Checkpoint 6 (May 27): Final Project Demo Video.
7. Demo (May 28): Final Project Demo (maybe Arya and Bea will be on campus). Visitors sit at the Touch device with the physical attachment and attempt peg-in-hole insertion with haptic feedback on, then off. Stretch goal: if fine-tuning is complete, fine-tuned policy results are shown alongside the baseline for all three VLA variants.
8. Final report (Jun 2): Theoretical derivation of impedance loop stability condition, experimental validation, hardware attachment design description, quantitative comparison of human performance across conditions, (bonus) pretrained SmolVLA baseline results, discussion of implications for VLA training pipelines. Stretch goal if achieved: fine-tuned policy evaluation comparing all three SmolVLA variants with analysis of what haptic-assisted demonstrations contribute to policy quality.

Checkpoint 1

Checkpoint 1 (May 14): Control and system dynamics drafted with relevant equations. Sketches of physical attachment ideas. For Checkpoint 1, our team first met with the teaching team to discuss our technical skills, project interests, and possible ideas before finalizing our project topic. We also evaluated the hardware and software compatibility of our laptops with the Touch device to ensure the system could be properly programmed and interfaced without technical limitations. This initial setup phase helped us confirm that our proposed project was feasible with the available resources and equipment.

Physical attachment ideas After determining the project direction, we began developing the background and design approach for the 3D-printed component. One of the objectives of our project is to improve the realism of the peg-in-hole task by optimizing the user’s haptic interaction with the Touch device. To achieve this, we decided to conduct a comparison study between two handle configurations: the default stylus provided with the Touch device and a custom peg-shaped handle designed to better replicate the feel of an actual peg-in-hole task.

To make the comparison more reliable and experimentally consistent, we also decided to add an extension to the default stylus design. This modification ensures that the distance between the actuators and the point of contact remains approximately equal for both handle configurations. By standardizing these dimensions, we can perform a more controlled comparison between the default stylus and the custom peg-like design while minimizing variability caused by geometry differences rather than the handle shape itself. At this stage, the main checkpoint goals that were completed include project topic selection, technical compatibility verification, establishment of the experimental comparison methodology, and the initial conceptual design of the 3D-printed handle extension. Future work will focus on finalizing the CAD model, manufacturing the components, integrating them with the Touch device, and testing user interaction and performance differences between the two handle designs.

Initial 2D design of the handle
and the inspiration behind it

Controls and System Dynamics

Assumptions:

1. (Isotropic) damping coefficients are each the same in all directions: B̅ = diag(b, b, b)
2. Negligible velocity in x & y directions. The primary motion is in the insertion axis (z).

Terms:

1. Device mass, m
2. Device damping [3x3], B̅
3. Rendered stiffness [3x3], K̅
4. External human input force in time-series, f,,human,,(t)
5. Touch actuator force outputted, f,,act,,
6. 3D position of the peg manipulated by gripper in the virtual environment, [x, y, z]

Equations of Motion:

The full 3D vector equation of motion is:

where f_act is the actuator force rendered by the impedance controller

The displacement vector is provided by the virtual environment (MuJoCo contact data).

Substituting f_act :

Transfer Function (Laplace Domain):

Taking the Laplace transform and treating f_human(t) as an external input:

Stability Condition:

The poles of the system are:

Oscillations occur when the poles have an imaginary part. Therefore the stability condition is:

Natural Frequency and Damping Ratio:

Referencing the standard 2nd order form:

Natural frequency:

Damping ratio: From comparison,

Substituting the stability bound K:

So as K decreases below the bound, ζ > 1 --> the system becomes overdamped.

For this project, we aim to operate close to the stability bound K̅ to ensure stability while rendering enough force for user feedback during collision with the peg hole rim (better haptic fidelity).

Checkpoint 2

Here you will write a few paragraphs about what you accomplished in the project so far. Include the checkpoint goals and describe which goals were met (and how), which were not (what were the challenges?), and any change of plans for the project based on what you learned. Include images and/or drawings where appropriate.

Example Video: https://www.youtube.com/watch?v=i_aLBql4Ufo