Haptic Hardness Detection with Teleoperated Two-Finger Gripper

Project team member(s): Abdul-Raheem Adeleke, Adarsh Anil Kumar, Yiyang Mu, Vishal Singh

Our project seeks to design a finger-controlled teleoperated gripper that not only allows the user to grab and hold an object from a distance but also provides valuable haptic feedback regarding the hardness attributes of the object. We utilize four hapkit devices, two of which are grippers and two are end handles into which the user can insert their fingers. Utilizing proportion derivative control, the user can feel the hardness of the object held by the grippers from force feedback.

Introduction

In this project, a user will use a finger-controlled teleoperated remote gripper that produces haptic feedback informing the user of the object's orientation and surface hardness. The motion of the user's finger on the leader will be used to teleoperate the gripper (follower) to grab an object. Through this device, the user will be able to feel the object from far away. The project will provide the user with the immersive experience of handling objects of different hardness such as a soft sponge or piece of wood. We will utilize four hapkit devices, two of which are grippers and two are end handles into which the user can insert their fingers. We will design and 3D print the parts for grippers and end handles that will be mounted on the hapkit devices. We will implement teleoperation control with Arduino codes.

The project is motivated by several key factors within the industry. By incorporating hardness detection, the gripper enables improved object handling, particularly in industries that deal with a wide range of materials and hardness levels. This enhances quality control processes by automating sorting, grading, and handling tasks based on hardness characteristics. Moreover, the project facilitates effective human-robot collaboration by providing operators with haptic feedback and the ability to remotely control the gripper. This collaboration streamlines tasks, improves productivity, and ensures safety by leveraging the strengths of both humans and robots. Additionally, the gripper's ability to adapt to different objects and environments based on hardness properties enhances customization and adaptability.

Background

Extensive research has been conducted on haptic feedback and teleoperation, leading to significant advancements in various domains. Teleoperation systems enable users to manipulate objects or control remote devices from a distance, providing a sense of presence and feedback. Haptic feedback plays a crucial role in enhancing user perception and performance in teleoperation tasks. Prior studies have laid the foundation for our research, which focuses on developing a finger-controlled teleoperated gripper with haptic feedback for hardness perception.

The hapkit platform, introduced in 2012 at Stanford University, is a 1-degree-of-freedom versatile tool for providing haptic feedback which we have utilized for our project. One notable study by Pierce [1] is driven by the potential of robotic teleoperation platforms to enhance human-robot interaction. They developed a wearable haptic device that provides bilateral control over a remote robot's gripper. This device is the first of its kind to offer comprehensive haptic feedback, including kinesthetic grip force, fingertip contact, pressure, and vibrotactile feedback. It enables intuitive control of the gipper's aperture through finger movements. By combining a geared DC motor and linear voice-coil actuators, the device delivers real-time haptic feedback to the operator. Our project is heavily inspired by this work.

Another relevant study explored the use of series elastic actuators in a finger-functional parallel gripper [2]. The researchers conducted experiments to demonstrate stable force-position control and studied pinching hardness of different objects. Their findings regarding gripper design, mechanism, and control serve as a valuable reference for our own teleoperated gripper design.

Park, Park, and Bae [3] introduced an innovative teleoperation interface integrated with an anthropomorphic teleoperated robot and a hand-wearable force-feedback system. Through two experiments involving ten participants, they evaluate the performance of tactile and kinesthetic finger feedback. The results demonstrate that providing force feedback to the user's finger is highly effective in teleoperation tasks. This study contributes valuable insights into haptic feedback mechanisms for teleoperation and inspires our own research to enhance user experience and performance in similar scenarios.

The paper by Bimbo [4] and Tay [5] highlights the application of teleoperation and haptic feedback in handling and hardness discrimination tasks. These studies demonstrate the feasibility and effectiveness of using teleoperation and force feedback to discriminate gripped objects based on their hardness properties.

In summary, previous research on haptics, including the hapkit platform and studies on teleoperation and hardness discrimination, provide valuable insights for our research on a finger-controlled teleoperated gripper with haptic feedback for hardness perception. By leveraging these advancements, we aim to enhance user experience, performance, and perception in teleoperation tasks involving objects with varying hardness characteristics.

Methods

Hardware design and implementation

The design of our teleoperated gripper was based around the existing hapkit paddle assembly, as depicted in Figure 1. Due to limitations in time and resources, we opted to build the gripper by making several attachments to the existing hapkit. Our goal was to provide users with enhanced maneuverability and a realistic grasping sensation. To achieve this, we incorporated two independent finger assemblies within the gripper, enabling users to manipulate the fingers of the remote grippers individually or collectively using their own two fingers. Additionally, parallel linkages were implemented to ensure consistent and strong grasping ability throughout the gripper.

Our project assembly comprised four hapkit paddle sub-assemblies, each associated with a dedicated microcontroller. For teleoperation, the left finger of the user's side gripper was connected to the corresponding finger of the remote gripper, and the same principle applied to the right side. The design of each of the four fingers was identical, except for their respective end effectors. The remote gripper's end effector drew inspiration from the fin-ray soft robotic gripper and was 3D printed using TPU-95 material. This material offered sufficient compliance to deform and conform to the shape of objects during grasping, while providing consistent data for hardness detection when tested on flat surfaces. The user's end gripper, on the other hand, featured an end effector 3D printed using PLA material, which incorporated large pockets to accommodate the user's index and thumb fingers.

To realize the parallel linkage assemblies for all the fingers, 3D printing was employed with PLA. The base structures for both the remote and user end grippers were laser-cut from clear acrylic. Furthermore, to ensure stability during gripper motion, the entire base assembly was affixed to a duron sheet equipped with suction cups at the bottom. This design approach allowed us to develop a teleoperated gripper that capitalizes on the existing hapkit while offering improved functionality and a realistic grasping experience. The use of different materials and 3D printing techniques facilitated the construction of flexible and robust finger assemblies. Additionally, the inclusion of parallel linkages and the integration of suction cups enhanced the gripper's performance and stability during operation. Please refer to Figure for a visual representation of our teleoperated gripper design.

System analysis and control

In our design, each gripper will be controlled by an end handle through teleoperation. We implement teleoperation using a similar approach as in Assignment 5. For each pair of the gripper and the handle, the gripper will be the follower while the handle will be the leader. The teleoperation requires the use of two hapkit boards (i.e., one for the leader and one for the follower) and two hapkits. Each hapkit will be driven by a motor powered and controlled by the controller board. Hence, the implementation of teleoperation is based on displacement measurement. As the user moves the handle, the gripper will track the handle. The handle will also track the gripper. The user will feel the haptic feedback (i.e., force) from the handle while the gripper is holding the object. The user can feel the hardness of the object via force feedback. The greater the force feedback, the harder the object is held by the grippers. Basically, each pair of gripper and handle uses one leader and follower code. The codes track the position of the leader and follower using PD controllers. We utilize the following equations for teleoperation:

We can then compute the torque produced by the motor from the force. During the teleoperation, the leader and follower controller boards will communicate the handle positions to achieve the position tracking. In addition, we tune the gain values (i.e., kp and kd) to make the system stable and noise-free. We also adjust the gain values to enhance the haptic experience of the user. We had a hard time tuning the control gains. Even with very low gain values, the hapkits were very unstable. We eventually realized that the instability was caused by inconsistent friction between the gripper parts. We modified the gripper design to reduce friction, resulting in much smoother and more stable teleoperation control. Table 1 shows the finalized gain values for each gripper and handle.

The functioning of our teleoperating two fingered gripper can be seen below:

Demonstration / application

After the appropriate gain values were obtained and the entire system was calibrated, we developed a user study to determine the efficacy of our project and its haptic feedback. Four objects of the same dimensions were identified to test the capability of our project to convey hardness detail to the user when utilizing the remote gripper to grab something. The four objects were: Yellow sponge, Soft foam, Hard foam and Wood (see Figure 4). This study was inspired from the paper by Zamani [6].

The user is initially allowed to play around with the tele operating gripper to get an idea of how it works and feels. Then the user is blindfolded and asked to rate the hardness of the four objects (presented in a random order) on a scale from 1 to 10 where 1 depicts the absence of any object to compress and 10 depicts using their own fingers to compress a rock-hard object. The rating for each object was then recorded and the responses of 11 users were analyzed for the same. It is to be noted that we had collected 18 user responses but due to the gripper malfunctioning in between one of the experiments as the user applied a lot of force causing the motor to start saturating randomly. Hence, around 7 responses were eliminated from analysis for lack of consistency.

A snipper of the user study where the user is grabbing a block of soft foam (one of our test pieces) can be seen below:

After the user study, we also encouraged the users to utilize the gripper to grab objects of different shapes such as a ping pong ball, a paper cup, a plastic bottle etc.

Results

The results from the user study were used to generate the following results. First, the averages of the estimated object stiffness were calculated. The results shown in figure x show on average the users were able to accurately distinguish the objects in the correct order of stiffness such that the sponge was the softest object, followed by the soft foam, hard foam, and wood respectively. This result matches the expected outcome.

Additionally, the standard deviation was taken for each of the estimated object stiffness values. The standard deviation stayed around 1.4 for the sponge, soft foam, and hard foam. This consistency shows that despite differences in how users estimated the object stiffness; the users overall picked values for stiffness that were within the same range. The standard deviation for the wood was slightly higher at 1.63. This increase can be due to the wood being the stiffest object. This means users would have a more difficult time gauging the upper bound stiffness of the wood so users could have chosen a wider range of values.

A trendline analysis of the ratings was also conducted for the four different materials which revealed important insights into the subjective perceptions of hardness among users. The trendlines demonstrated distinct patterns for each material. Sponge and soft foam displayed relatively higher variability compared to hard foam and wood, suggesting a wider range of tactile experiences and sensitivity to hardness differences.

The observed switching patterns and changes in hardness rankings across materials indicated dynamic perceptions of hardness characteristics. Some users consistently perceived sponge or soft foam as softer materials compared to hard foam and wood. However, there were users who displayed varying rankings, indicating different sensitivities to hardness variations or subjective interpretations of hardness.

The hardness ratings provided valuable insights into the perceived hardness characteristics of the materials. Factors such as surface texture, material composition, and haptic feedback likely influenced the hardness ratings. Additionally, task requirements and individual preferences may have contributed to variations in the hardness perceptions. Overall the results show that the gripper system fulfilled its purpose of allowing the user to feel the stiffness of a range of objects accurately based on the complete results of what users on average found to be softer and harder.

Future Work

It is important to acknowledge some limitations of this study. The sample size included 11 users, which may not fully represent the entire user population. Future studies with larger and more diverse user groups could provide a more comprehensive understanding of hardness perceptions. Additionally, exploring additional factors such as material roughness or compliance in the rating process may enhance the evaluation of hardness characteristics. It is also important to upgrade the mechanism so that we can get more accurate and consistent results for hardness perception.

After understanding the limitations of our current applications and compiling feedback from the users we have formulated a series of updates and future installments we want to include into our project. One of the major initial updates would be to reduce the overall mechanical friction in the project through the implementation of bearings and shaft collars everywhere possible. In order to read more consistent data with respect to hardness, we are also planning to make the end effectors out of a mixture of TPU-95 and PLA to get both non-compliant consistency and complaint handling. This can be executed by developing an adaptive compliant gripper finger.

From a control point of view, the attachment of FSR sensors to the remote gripper end and actively adjusting the force-feedback based on the readings on the FSR sensors will give us a better understanding of the hardness of materials. Reducing the surface area of the grippers is another huge step as it will help clearly define hardness as opposed to compressibility of an object. By integrating additional sensory cues such as vibrations, the haptic experience can be enriched, providing a more comprehensive and immersive user experience. Exploring the optimal combinations of force and position feedback and studying their impact on user perception presents an intriguing avenue for research. Furthermore, the entire system can be optimized and made adaptive to different users in real-time, further improving its effectiveness. The interaction between humans and robots is a crucial factor to consider, particularly when using a teleoperated gripper where the human controls the gripper remotely. The feedback from the gripper can assist humans in making informed decisions, thereby opening up opportunities for improving the user interface. Additionally, studying the effects of latency and communication delays on the system's performance can lead to enhancements in its overall effectiveness.

A promising future direction is the combination of haptic feedback with visual cues to create a more comprehensive perception for users, simulating the sensation of gripping an object. A thorough investigation can be conducted to compare the effects of visuals versus actual haptic feedback, followed by finding an optimal balance between the two. This will facilitate the development of devices that perform well even in environments with external factors or noises. By employing optical sensors at the grippers in addition to force sensors, the shape of an object can be detected, providing initial feedback to the user about the type of object they are gripping. Proximity sensors can assist in calculating the distance between the object and the gripper. Moreover, adaptive control algorithms can dynamically adjust the grip force based on the object's hardness, further enhancing the system's performance.

The incorporation of haptic hardness detection in the medical field holds great potential across various healthcare applications. It can enable healthcare professionals to perceive the hardness of objects they interact with, assess tissue or organ characteristics, and create more realistic training environments for medical practitioners.

Acknowledgments

We would like to express our deepest gratitude to Professor Allison Okamura for her invaluable guidance and mentorship throughout our exploration of various haptic topics. Her expertise and dedication have played a crucial role in establishing a strong foundation for the field of haptics and inspiring us to pursue this project. We are truly grateful for the opportunity to work under her supervision.

We would also like to extend our heartfelt appreciation to the teaching assistants of the class, Connor Yakp, Dane Brouwer and Elizabeth Danielle Vasquez. Their consistent support, insightful feedback, and tireless efforts have been instrumental in shaping our project and enhancing our understanding of haptic technologies. Their guidance has been invaluable in navigating challenges and ensuring the success of our endeavors.

Additionally, we would like to thank the Product Realization Lab (PRL) for providing the necessary resources and facilities that have enabled us to bring our project to fruition. Their assistance and cooperation have been crucial in the development and implementation of our work.

Lastly, we would like to express our gratitude to all the individuals who contributed in various ways, directly or indirectly, to the completion of this report. Their support, encouragement, and valuable insights have greatly enriched our learning experience and project outcomes. We are sincerely thankful for all the assistance and contributions received throughout this endeavor, as they have significantly impacted our journey in the field of haptics.

Files

Link to the CAD files: https://drive.google.com/file/d/1coG5zSD3HV0dqi9tD4Mhlm0DFbzfVKKA/view?usp=sharing

Link to a PDF of the design sketches: https://drive.google.com/file/d/1xqhecaE5kpmS95TcnGTAeL5ekHbgXSDW/view?usp=sharing

Link to Arduino Code: https://drive.google.com/drive/folders/1Vb9vLyOAUpTe6txW_zEpYdUHf4w4SYuI?usp=sharing

Link to Simple BOM: https://docs.google.com/spreadsheets/d/1t51e2mkS_w6mbPMdkDoTfaVuCfYimGt59nug0jwXiLA/edit?usp=sharing

References

[1] Pierce, R. M., Fedalei, E. A., and Kuchenbecker, K. J. (2014). "A wearable device for controlling a robot gripper with fingertip contact, pressure, vibrotactile, and grip force feedback." In 2014 IEEE Haptics Symposium (HAPTICS), pp. 19-25. doi: 10.1109/HAPTICS.2014.6775428. https://ieeexplore.ieee.org/abstract/document/6775428

[2] Wang, Y., Zhou, K., Ma, S., & Xu, Q. (2019). "SEA-Based Humanoid Finger-Functional Parallel Gripper With Two Actuators: PG2 Gripper." IEEE Access, 7, 168083-168093. https://ieeexplore.ieee.org/abstract/document/9989427

[3] Park, S., Park, Y., & Bae, J. (2022). "Performance evaluation of a tactile and kinesthetic finger feedback system for teleoperation." Mechatronics, 87, 102898. https://doi.org/10.1016/j.mechatronics.2022.102898

[4] J. Bimbo, C. Pacchierotti, M. Aggravi, N. Tsagarakis and D. Prattichizzo, "Teleoperation in cluttered environments using wearable haptic feedback," 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, 2017, pp. 3401-3408. https://ieeexplore.ieee.org/abstract/document/8206180

[5] Tay, T.T., & Wong, K. (2010). "Creating and measuring relative hardness feeling via master-slave haptic gripper." Yanbu Journal of Engineering and Science, 1(1), 12-25. https://yjes.scholasticahq.com/article/23735.pdf

[6] Zamani, N., & Culbertson, H. (2022). "Combining Haptic Augmented Reality with a Stylus-Based Encountered-Type Display to Modify Perceived Hardness." 2022 IEEE Haptics Symposium (HAPTICS), Santa Barbara, CA, USA, pp. 1-7. https://ieeexplore.ieee.org/abstract/document/9765601

---

Appendix: Project Checkpoints

Checkpoint 1

Tasks Completed: So far for the project, we have established the scope of our work. We plan to develop a user-controlled teleoperated robotic gripper. The gripper will have haptic feedback so that the user can tell the hardness and orientation of an object in the gripper. An additional task will be that the gripper can detect if a person is pulling on the grasped object and it will use haptic feedback to indicate the user to release the object.

We submitted the project proposal which contains the topic and motivation, previous work, a summary of the previous work, and the project plan. The project topic is explained in the previous section. The motivation behind the project was several key factors in industry use of robotics such as object handling, object sorting, and human-robot interaction. In our research of previous work, we found research papers that covered the topics such as the designs of simple robotic grippers and the design of a hand-wearable force feedback system to control a robotic gripper. These papers were very applicable to our project and will be used as a reference for our designs. Lastly, we developed our project plan which lays out how we will go through the development cycle of the project. Below is a visual timeline of our plan.

Action Items: As of now, we are still working on getting the preliminary design finalized. Our team is currently working on the CAD models of the hardware and the Arduino code of the teleoperation simultaneously. One of our team members is focusing on CAD models, whereas the other members are working on the code. We will implement teleoperation of our design using a similar approach as in Assignment 5. We will need two laptops, each of which will run the leader/follower code. The teleoperation will require the use of four hapkit boards (i.e., two for the leader and two for the follower) and four hapkits. Each of the hapkit handles will be driven by a motor, where each board will power and control the motors. The displacement will be measured using the existing magneto-resistive sensors on each hapkit board. Then, the follower will be controlled by a teleoperation code similar to that in Assignment 5. Once the design is finalized, we will continue working on the scheduled tasks such as completing the Arduino code, testing the teleoperation, and setup hardness differentiation. Overall, we will continue with the plan as written and don’t intend to make any changes.

Preliminary CAD: In terms of finalizing the CAD design, we are still looking at inventory in order to size the shaft appropriately for the gripper using shoulder screws (deciding between using a bearing or not). In terms of 3d printing, we have modified the existing handle and the motor base which will be printed using PLA. The assembly base will be made using duron as well as the links to the gripper. We are still exploring the potential of using a free hinge gripper versus a parallel linked gripper.

In terms of design, the leader and follower are identical except for the fingers of the grippers. The leader has pockets in its gripper finger to accommodate the human index and thumb finger and the follower’s grippers do not. The gripper fingers will also be 3D printed using PLA.

Checkpoint 2

Tasks Completed: We were able to finalize the overall CAD and 3D print/ Laser cut all the parts necessary for our project. We also accrued all the hardware we need for the project including bearings, shoulder screws etc. Before the checkpoint, we were able to assemble one half of the final assembly i.e one gripping end for the Leader and Follower respectively.

In terms of design changes, we decided to use the existing hapkit assembly and only change the hapkit circle with addition to the inclusion of our grippers. We were able to implement a parallel linkage system with a compliant gripper finger that can grip objects of any shape. The key feature of our project is the way the user can independently control each finger of the gripper using either their index finger, thumb finger or both together.

We were able to test out the initial validity of the Leader and Follower code from Assignment 5. But we were not able to implement it the way we want which we are currently working on. We have completely taken out the possibility of implementing a handing off feature on our project due to limitations in time.

In Progress Tasks: We are currently making updates to the Arduino code that is running on the gripper and finger controlled hapkit. The issue we encountered was that the current code allowed for teleoperation of the gripper using the controller hapkit, however no force feedback was felt in the controller hapkit. This meant that the user would not be able to feel if the gripper hit an object which would make it difficult for the user to accomplish the primary task of the project. In troubleshooting this issue we determined that we will most likely have to do recalibrations for some of the values used for the force and torque calculations used in the controller hapkit code.

Action Items: We need to test and implement a completely functional follower and leader code with well calibrated feedback such that the user can feel the resistance for the teleoperated gripper when it is handling an object. We also need to make a finalized list of objects we want the test users to feel and guess what they are. An according calibration of our gripper to these different objects is also needed.