TouchWorld: Haptic Exploration of AI-Generated 3D Environments

Project team member(s): Aytan Sadirova, Herron Ha-Vi Nguyen, Marie Imad, Sukeerth Ramkumar

Give a one-paragraph description/summary of the project, which includes information like the motivation for the project, the goals of the project, and the resulting implementation and success of the project.

Introduction

Explain the motivation for your project in terms of the educational objectives and why your haptic device is an appropriate approach.

Background

Explain the relevant prior work in the field of haptics and provide references. These will likely be different from the references in your project proposal. Make sure to do a thorough literature search on relevant haptic devices/application.

Methods

Provide a detailed description of your project, such that another student from the class could generally re-create your project/experiment from the report if necessary. (You don't need to document every screw, but the design should be clear.) Add images and videos as needed to support the description. You can refer to downloadable drawings and code in the "Files" section (later). You should divide this section into subsections, which can vary depending on your particular project. Here is an example set of subsections:

Hardware Design and Implementation

System Analysis and Control

Demonstration / Application

Results

Describe the results, which may include qualitative responses from users at the open house.

Future Work

Describe how your system could be tested (e.g., through experiments if you have not already done so), how it can be improved, and how it might be applied.

Acknowledgments

Here you can list any individuals or groups who helped you with your project. (e.g., another student in the class, a course assistant, or an especially helpful PRL TA). Optional, so delete this section if you aren't using it.

Files

Code and drawings should be linked here. You should be able to upload these using the Attach command. If you aren't willing to share these data on a public site, please discuss with the instructor. Also, in this section include a link to a file with a list of major components and their approximate costs.

References

List the referenced literature, websites, etc. here.

---

Appendix: Project Checkpoints

Checkpoint 1

Got it, so the checkpoint text itself is fine, but the overall wiki page structure needs to be set up correctly first. Here's the full wiki page template you should have, with your checkpoint already in the appendix at the bottom:

Attach:representative_image.png Δ Representative image caption.
Second line of caption if needed.

TouchWorld: haptic exploration of AI-generated 3D environments

Summary

Write your final abstract here (under 150 words). Describe what the project is, why it matters, and what you built.

Introduction

Explain the motivation for your project in terms of the educational objectives and why your haptic device is an appropriate approach.

Background

Explain relevant prior work in haptics and provide references.

Methods

Provide a detailed description of your project.

Hardware setup

Software pipeline

Force rendering

Results

Describe results, including qualitative responses from users at the open house.

Future work

Describe how the system could be tested, improved, and applied.

Acknowledgments

We would like to thank our course instructor Allison Okamura, and the PhD student at CHARM Lab who shared their Unity haptics prototype with our team.

Files

Links to code and drawings. List of major components and approximate costs.

References

List referenced literature and websites here.

---

Appendix: Project checkpoints

Checkpoint 1

Attach:representative_image.png Δ Representative image caption.
Second line of caption if needed.

TouchWorld: haptic exploration of AI-generated 3D environments

Summary

Write your final abstract here (under 150 words). Describe what the project is, why it matters, and what you built.

Introduction

Explain the motivation for your project in terms of the educational objectives and why your haptic device is an appropriate approach.

Background

Explain relevant prior work in haptics and provide references.

Methods

Provide a detailed description of your project.

Hardware setup

Software pipeline

Force rendering

Results

Describe results, including qualitative responses from users at the open house.

Future work

Describe how the system could be tested, improved, and applied.

Acknowledgments

We would like to thank our course instructor Allison Okamura, and the PhD student at CHARM Lab who shared their Unity haptics prototype with our team.

Files

Links to code and drawings. List of major components and approximate costs.

References

List referenced literature and websites here.

---

Appendix: Project checkpoints

Checkpoint 1

What we accomplished

We explored World Labs / Marble as our source for 3D Gaussian Splat scene reconstruction. Marble exports meshes in either OpenGL or OpenCV coordinate conventions (both right-handed). Unity uses a left-handed coordinate system and can import both, provided we account for the necessary pose transformations. Meshes are exported in GLB format as surface meshes (2D triangles in 3D space), storing indices, vertices, normals, and optional UVs, colors, and materials. A key constraint we identified is that most Marble meshes are not watertight, meaning the boundary between inside and outside the mesh is not explicitly defined. This directly shapes our haptic rendering approach.

We met with a PhD student at CHARM Lab who demonstrated an early Unity prototype integrating LLM-driven haptic interaction with 3D objects. She shared her Unity package, README, and backend dependency files, which we are using as a development reference. Relevant resources include the 3D Systems Haptics Direct for Unity asset and the OpenHaptics driver suite, both of which we are actively evaluating.

We are also investigating how to bridge our force dynamics model with the Unity 3D visual world. The haptic loop must run at 1 kHz as required by the Touch device, while Unity's render loop runs at 60 to 120 Hz. We are evaluating two approaches: a native C++ plugin (.dll) that wraps OpenHaptics and exposes force and position state to Unity via P/Invoke, and a UDP socket bridge where the haptic process streams position and force vectors over localhost to Unity in real time.

width=40% Figure 1: Coordinate frame conventions for PyTorch3D, OpenCV, OpenGL, and Unity. Marble exports use OpenGL or OpenCV conventions, requiring pose transformation on import to Unity's left-handed coordinate system.

width=40% Figure 2: Full system overview covering scene capture, Unity visual layer, collision pipeline, force-to-Unity bridge, and hardware output.

 Forces in the OpenHaptics SDK

We extracted and read the source files inside Developer/3.5.0/examples/HL/graphics/HapticMaterials/HapticMaterials.cpp and Developer/3.5.0/Quickhaptics/examples/ShapeDepthFeedback/. These files revealed how OpenHaptics computes and delivers forces to the device. Three forces are automatically computed at each servo tick (1 kHz) when the stylus contacts a surface.

 1. Normal force (contact resistance):  

Fnormal = k · δx · n

δx is the penetration depth — how far the stylus tip has entered the virtual surface. n is the surface normal (perpendicular direction pointing outward). The force always pushes the stylus back out. In the SDK this is controlled by HL STIFFNESS (k ∈ [0, 1]): k = 1.0 gives maximum resistance (stone), k = 0.3 gives a soft response (fabric).

 2. Damping force (velocity resistance):  

Fdamp = −b · x'

x' is the velocity of the stylus. This force opposes motion and models energy dissipation in the material - a high damping value makes the surface feel viscous or sticky (fabric, mud), while a low value gives a clean sharp contact (stone, metal). Controlled by HL DAMPING (b ∈ [0, 1]).

 3. Friction forces (surface drag):  

Ffriction = μs · Fn (static) and Ffriction = μd · Fn (dynamic)

Static friction resists the start of lateral motion; dynamic friction resists ongoing sliding across the surface. These are set independently via HL STATIC FRICTION and HL DYNAMIC FRICTION.

 4. Friction forces (surface drag):  

Ftexture(t) = A · sin(2πf t)

A sinusoidal vibration superimposed on the contact force, delivered through the device motors. A controls the intensity of the texture sensation and f controls the perceived granularity: high frequency (200 Hz) simulates smooth hard surfaces like polished stone; low frequency (30 Hz) simulates coarse soft surfaces like fabric.

The full combined force rendered at each timestep is therefore:

F (t) = k · δx · n − b · x' + A sin(2πf t) · n

In the SDK, all four parameters are set per surface with a single function call:

hlMaterialf(HL_FRONT_AND_BACK, HL_STIFFNESS, k ); hlMaterialf(HL_FRONT_AND_BACK, HL_DAMPING, b ); hlMaterialf(HL_FRONT_AND_BACK, HL_STATIC_FRICTION, mu_s); hlMaterialf(HL_FRONT_AND_BACK, HL_DYNAMIC_FRICTION, mu_d);

This means that to switch from a stone surface to a fabric surface, we only change these four values - the rest of the rendering pipeline stays identical. This is exactly the material mapping mechanism we will implement in Unity for the full Bamboo Courtyard scene.

Checkpoint goals

1. Identified 3D scene reconstruction workflow (World Labs to Marble to GLB mesh export). First workflow is subject to change as we explore further.
2. Established coordinate frame conventions (OpenGL/OpenCV to Unity transformation)
3. Connected with PhD student at CHARM Lab and obtained working Unity haptics prototype for reference
4. Haptic rendering algorithm: in progress. We are evaluating BVH, AABB trees, and Oct-tree approaches for mesh collision. CS248A graphics notes are a useful reference.
5. Smooth surface haptic feedback: planned. A smoothing function at the triangle mesh level is required and will be a focus for Checkpoint 2.
6. Force-to-Unity communication bridge: in progress. Architecture is under active evaluation.

width=40%
Figure 3: Collision pipeline detail showing BVH construction, contact query, normal interpolation, and spring damper force computation.

width=40%
Figure 4: Runtime architecture showing the 1 kHz haptic thread and 60 Hz Unity thread running in parallel with position and mesh sync between them.

Challenges

The primary technical challenge is that Marble meshes are non-watertight, which complicates standard inside/outside collision detection. We are researching haptic rendering algorithms compatible with open surface meshes using OpenHaptics. Speech-based LLM input is likely too noisy for demo day conditions. We are moving toward a push-to-talk interaction model as a more reliable alternative.

Change of plans

We are shifting from a Unity-native collision approach to a custom haptic rendering pipeline built to handle non-watertight mesh geometry. Resolving the force-to-Unity communication architecture is now a defined goal for Checkpoint 2.

Checkpoint 1

Checkpoint 2