Cutting vegetables with haptic knife
and visual rendering.

I Can't Believe It's Not Cutting!

Project team members: Jacob Azoulay, William Choi, Ambikaa Jaggi, Renee Nguyen

The goal of the project was to create a haptic display to simulate the experience of chopping a vegetable through a two degree of freedom system. The display has two hapkits mounted on the back wall of a box: a horizontal hapkit controlled by "the carrot" that moves along the cutting board in order to position the vegetable under the knife, and a vertical hapkit with a 3D printed knife handle for cutting. The display was accompanied by a real time visual and auditory rendering. Overall users enjoyed the experience, and thus the project was deemed sucessful.

Introduction

Our team would like to study how effectively you could train a user to cut virtual fruits or vegetables. The project is aimed at either those with lack of kitchen experience or those who need to rehabilitate their hand movements. Another aim for this project is to gauge how realistic the cutting experience is in the haptic space compared to real space using a PD-controlled simulation. In order to adjust for the weight of the knife the equations of motions will inform the necessary controllers to overcome the feeling of gravity when the knife makes contact with the object.

Project team members: Ambikaa Jaggi, William Choi, Renee Nguyen, Jacob Azoulay (left to right)

Background

In Zhang et al. [1], the authors contextualize their research with the knowledge that chopping food is an inherently difficult task due to significant variability in shape, hardness, amongst other material properties. This warrants the need to learn different chopping techniques as well as safety practices for both the user and the equipment. This paper focuses specifically on training a robotic arm using neural networks to learn to identify and perform appropriate chopping techniques using force and torque feedback. The authors have developed a clear outline of what constitutes a specific chopping action, which would help us develop what spring stiffness, damping, and inertia correlate to each item we plan on simulating the chopping experience form, as well as what motion data correlates to relevant checkpoints within the chopping motion.

In Okamura et al. [2], the authors develop the “Haptic Scissors” device that allows for haptic rendering of cutting in a virtual environment. This two degree of freedom (DoF) device mimics realistic cutting sensations by using an algorithm based on haptic data recorded from an actual cutting experience and a different algorithm developed analytically. In the model, the scissors first detect whether they have come in contact with an object to be cut using trigonometry. Then the cutting algorithms take effect and produce a force. Depending on the material, a different force is applied to mimic the object dynamics of being cut. We can use some of the force models to apply for our cutting of different objects.

In G. Srimathveeravalli and K. Thenkurussi [3], the authors explored the use of haptic devices as training tools for complex motor skills. While learning a new skill, the virtual environment is leveraged to augment the user’s sense of proprioception to accelerate the learning process. The authors designed a test to teach the skill of handwriting. They compared the efficacy of force feedback vs. PD position feedback. Both methods were contextualized in comparison to a control method of no feedback. Users were presented shapes and asked to recreate them in the virtual environment using a pen-shaped end effector. The haptic device implemented one of the three aforementioned control schemes to aid the user. After drawing the shapes with assistance, the users were asked to recreate the shapes without assistance. Performance in this “recall” test was used as the barometer to determine the effectiveness of the training method. In conclusion, the authors determined that force feedback was the most effective training modality.

Methods

Hardware design and implementation

We propose to design a 2 DoF haptic device that will mimic the motions of cutting various foods along a virtual cutting board. One hapkit will be used to apply the necessary stiffness to the knife to simulate cutting an object. In order to achieve this, the handle of the hapkit will be replaced with a 3D printed knife handle. A second hapkit will also be motorized and used to simulate pushing an object in virtual space to align with the cutting edge of the knife. The motor will apply a constant force to ensure restoration of the hapkit to the neutral position for the next user. A duron box will be used to house and hide the internals in an effort to maximize the realism of the simulation.

Annotated hapkit assembly design

We would like to both determine the range of cutting board object stiffnesses that can be realistically rendered and characterize the naturalness of a 1 DoF mechanism to simulate the cutting motion.

Bill of Materials:

• 2 Hapkits
• 1 3D Printed Knife
• 1 PVC Pipe
• 2 18 x 24 sheets of Duron
• 1 Computer with Rendering Environment
• 3 3 x 7 black fabric for opening covers

Haptic Rendering

The haptic feedback of the entire system can be decomposed into the sensation from the knife and the sensation from the carrot. While not physically coupled, the dynamic haptic feedback of the knife is influenced by the state of the carrot handle and vice versa.

The haptic design of a cutting sensation was motivated by the true sensation felt when chopping a carrot. Prior to executing a cut, a user holding a knife experiences no haptic feedback from the knife since it is in free space. As a knife makes contact with the surface of the carrot, an immediate wall like force is felt. Once enough force is applied by the user, the knife overcomes a force threshold of the carrot and abruptly slices through the remainder of the carrot. The knife is finally stopped by the hard surface of the cutting board.

To mimic this sensation, an Arduino file that renders a carrot as a stiff spring-like wall followed by a constant drag force was developed. There are four state spaces the knife can move in. First, when the knife is raised completely, there is zero haptic force exerted onto the handle. This resembles free space in which the user is not making any contact with the carrot or cutting board. Once the user lowers the knife to a specified depth, the user feels the surface of the carrot. The carrot is rendered in two parts. The first is a very stiff wall that has a small width. This is meant to simulate the initial feeling of cutting into the carrot where the user must apply a larger force than required to cut into an object. Once inside the carrot they have passed this stiff wall and a constant force is applied as long as the user is pushing the handle downwards. This is meant to simulate the inside of the carrot. Finally, the knife makes contact with the cutting board, represented by the physical stop block under the knife.

To further aid realism the force of cutting into the carrot is only applied in one direction. The user is free to bring the knife back up even if they have not cut through the entire carrot and they will not feel any force feedback.

Lastly, the knife cutting sensation is only rendered if the knife is able to perform a valid cut. In other words, if there is any part of the carrot directly below the knife, then the user will feel the haptic feedback. Otherwise, the user will only feel the cutting board. Information about the carrot position is sent to the knife handle within the Processing script.

Example of a valid cut position.

The purpose of the carrot handle is to control the position of the virtual carrot. Because the handle was not rigidly attached to the hapkit, a small constant restorative force was applied in order for the hapkit handle to always maintain contact with the carrot stick. The force was minimal enough to not be felt by the user, but large enough to effectively track the carrot stick position.

To best mimic a realistic sensation two constraints on the motion of the carrot were imposed:

Both conditions require sending information on the state of the knife to the carrot. To implement the first constraint, when the knife handle is down and the carrot attempts to slide into the knife from either end, a virtual stiff wall is rendered at the location of the knife. For instance, if the knife handle is down and the carrot is to the left of the handle, the user is free to slide the carrot up until when the rightmost tip of the knife makes contact with the knife.

The second constraint is imposed by instantaneously modeling the carrot as a spring the moment the knife begins to slice through it. The position the carrot is in when the knife makes contact is stored as the carrots natural resting position. Any perturbations to the right or left will be restored back to the center.

Visual rendering

The physical haptic feedback was accompanied by a visual and auditory rendering which is coded using Processing. An image of a carrot, an image of a knife, a background image of a chopping board, and a one second audio file of a chopping sound are leveraged by the visualization. The Arduinos are able to write their x position to the serial monitor, which Processing is then able to read and use to display graphics in their correct locations.

The knife rendering, like the physical device, is only able to translate up and down, and is positioned off-centered to the right. As the hapkit knife is moved, the knife on screen tracks this movement. Likewise, the on screen carrot tracks the physical carrot as well and is constrained to only translate horizontally.

Starting position of the rendering

Once the knife is raised and the carrot is slid underneath the knife, the user is in a valid state to execute a cut. Once the knife completes the cut, the slice of the carrot visually falls off onto the right side of the cutting board. In code, the carrot image is essentially cropped at the point the knife makes contact with the carrot and each new slice image is appended to an array. The remaining carrot image that is controlled by the carrot hapkit is updated as well to represent the cut carrot. The cut slices are randomly scattered onto the cutting board. As each slice is executed, a chopping noise is also played.

Single carrot cut

To account for the physical constraints imposed on the carrot, visual haptic illusions are implemented.

Both visual tricks reinforce the sensation of a physical hard surface.

Multiple carrot cuts

System analysis and control

Dynamical Analysis: Attach:KHD2.pdf

Demonstration / application

Attach:carrot_cutting_demo.MOV

Results

Overall the haptics open house was successful as users really enjoyed interacting with the display. The 2 DoF system coupled with a higher than expected production value virtual rendering caused both surprise and delight. However, continuously running our hapkit motors for such an extended period of time brought to light certain design challenges. The motor associated with the carrot motion started overheating and eventually stopped providing the restorative force. We believe this was due to both running for a prolonged period of time, significant stall due to multiple user tests, the weight of 5 3D-printed handles to prevent users from jamming the carrot, and the magnitude of the restorative force required for the hapkit to accurately track the carrot. Thus, an approach previously discarded as detracting from the haptic illusion was implemented during the open house. A twist tie was duct-taped to the carrot and the hapkit, and the motors were unplugged, thus enabling users to move the carrot freely without any opposing force. The Arduino board remained connected to the hapkit to sense the position of the carrot. The tradeoff of this design was a compromised sense of realism. Due to the introduction of a flexible joint, the hapkit could not track the position of the carrot without backlash, even at low speeds. Furthermore, the user could physically move the carrot through the knife without force feedback. Finally, due to the rotational motion of the hapkit, attaching the carrot to the hapkit through a single joint meant that the carrot had to rotate to move the handle. Fortunately, none of these drawbacks were noticeable to the average user. Thus, despite the hardware setback, the haptic display met all goals and was deemed successful.

Future Work

Throughout the build process, we generated several ideas to improve the design of the system. However, due to time constraints, we were not able to implement all of them in the showcase iteration of the device. Checking out other projects during the showcase also inspired other design ideas we would like to implement in the future. The following are our goals for continued development of this device:

• Incorporate more degrees of freedom into the knife handle
• Implement a rack and pinion mechanism for the hapkit-carrot interface
• Expand the simulation to a greater variety of fruits and vegetables
• Enable manipulation of the cutting object in multiple degrees of freedom

The first two goals are aimed at improving the realism of the device. In order to allow users move the knife in a more ergonomic and natural manner, the knife must be able to move in 3 DOF within a plane. While users attested that the current 1 DOF design sufficiently approximates the motion of a knife, rotating the knife about a single point is an usual and unrealistic way to manipulate a knife. Typically there is some combination of rotation and translation in the cutting motion. Additionally, we would redesign the carrot-handle interface to a rack and pinion joint. The rack and pinion would allow tracking with minimal backlash and no restorative force, address the main concerns of both the flexible joint and detached design solutions. Finally, the rack and pinion converts rotational hapkit motion to linear carrot motion, solving another pathology of the flexible joint solution.

The third and fourth goals address the scope of the design. In order to provide a comprehensive overview of knife skills, the device should expose users to a variety of different objects with different haptic sensations. Additionally, the device should allow for a more complex orientation of the cutting object to allow more advanced users to learn different types of cuts.

Files

Code and CAD models are available on GitHub

References

[1] K. Zhang, M. Sharma, M. Veloso and O. Kroemer, "Leveraging Multimodal Haptic Sensory Data for Robust Cutting," 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), 2019, pp. 409-416, doi: 10.1109/Humanoids43949.2019.9035073. <https://ieeexplore.ieee.org/document/9035073>

[2] A. M. Okamura, R. J. Webster, J. T. Nolin, K. W. Johnson and H. Jafry, "The haptic scissors: cutting in virtual environments," 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422), 2003, pp. 828-833 vol.1, doi: 10.1109/ROBOT.2003.1241696. <https://ieeexplore.ieee.org/document/1241696>

[3] G. Srimathveeravalli and K. Thenkurussi, "Motor skill training assistance using haptic attributes," First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference, 2005, pp. 452-457, doi: 10.1109/WHC.2005.96. <https://ieeexplore.ieee.org/document/1406969>

Appendix: Project Checkpoints

Checkpoint 1

Current Hardware Setup State:

3D Printed knife handle for chopping hapkit:

Cardboard Prototype of Black Box Setup:

Dimensioned Box for Laser Cutting:

Current Haptics Code:

An Arduino file that renders a cutting board as a stiff wall as well as a carrot (as a stiff wall and constant force) has been created and tested. The code simulates the feeling of cutting into a carrot after the hapkit handle has been moved passed some point. The carrot is rendered in two parts. The first is a very stiff wall that has a small width. This is meant to simulate the initial feeling of cutting into the carrot where the user must apply a larger force than required to cut into an object. Once inside the carrot they have passed this stiff wall and a constant force is applied as long as the user is pushing the handle downwards. This is meant to simulate the inside of the carrot.

Once the user has cut through the whole carrot they will feel a stiff wall similar to the homework renderings when the handle is at the neutral (xh = 0) position. This is meant to simulate the cutting board. However due to the added lever arm of the knife handle the cutting board feels more akin to a spring than a wall. So this element may be replaced with a physical block in the setup so that the user is unable to move the knife past the neutral position.

To further aid realism the force of cutting into the carrot is only applied in one direction. Meaning the user is free to bring the knife back up even if they have not cut through the entire carrot and they will not feel any force feedback.

Current Virtual Rendering State:

Carrot cut via mouse clicks in Processing: Attach:virtual_rendering.mov

The virtual environment has a carrot placed on a chopping board, and whether a mouse is clicked, a chop along the carrot is made at that x position. Users are constrained to not be able to chop previously cut parts or beyond the edge of the chopping board to ensure greater visual realism. One of the greatest challenges with the rendering was adding realism to the chopped carrot pieces. By using a series of rotation and translation transformations, we were able to achieve the look of chopped pieces resting upon each other, with some occasional breaks in the illusion depending upon the size of the chop by the user.

While not part of our project plan's first checkpoint, the virtual environment rendering was experimented with early as we were unsure of the production value we should strive for and the difficulty of achieving that. However, after speaking with Dr. Okamura, we found that the realism of the virtual environment was sufficient. Thus next steps would entail moving away from a mouse click triggered chop and connecting the processing code to read the position data from the Arduino code linked to the hapkits.

Current Physical Design:

In order to improve the haptic illusion of the cutting a real carrot, we have designed a box to house the hardware "internals." This includes the hapkit, supporting structures, and power cables. Ideally, the user should only be able to see and feel the knife handle and the carrot. As shown in the figure above, we began the design process with a cardboard prototype to scope out rough dimensions and other required functionalities. In testing the rendering on the cardboard prototype, we noted the following physical requirements:

• The mounted surface must be sufficiently rigid to support the cutting force
• The through hole for the carrot must be a low friction interface so as not to render spurious forces
• The carrot must be guided in some way to ensure it would contact the proper hapkit upon insertion
• The knife needs a hard stop at the cutting board surface to increase realism
• The box must be as short as possible in the width dimension to increase realism

These requirements were incorporate into the CAD model of the box design. We plan to laser cut and assemble the box out of duron for the combination of its low price point, structural properties, and opaque color. The box will feature a slotted mounting plate for adjustable hapkit positioning, access holes for wiring, and a circular insertion point for the carrot. The box will have two open edges for access to the hardware internals. However, a two-sided cover piece (shown in the figure) will fit atop the box to shield the internals from view when the system is in use. When standing, the user will only see the knife handle and the carrot. Beneath the knife handle, there is a hardstop to recreate the effect of chopping down on a cutting board.

Checkpoint 2

Current Physical Design:

Design requirements and corresponding improvements from checkpoint 1 were as follows:

• The mounted surface must be sufficiently rigid to support the cutting force (laser cut box from duron)
• The through-hole for the carrot must be a low friction interface so as not to render spurious forces*
• The carrot must be guided in some way to ensure it would contact the proper hapkit upon insertion*
• The knife needs a hard stop at the cutting board surface to increase realism (block added)
• The box must be as short as possible in the width dimension to increase realism (see figure)

To address the rigidity concern, we laser cut and assembled the entire housing out of duron. The box was carefully dimensioned such that height of the carrot aligned with that of the cutting board. Slots were designed around the hapkit mounting holes to allow for fine tuning of the relative positions of the hapkits and the carrot. The total width of the box was selected to minimize the distance between the carrot and the knife. In the final design, the insertion point of the carrot into the box was about 8" from the knife blade. This means that the user holds the carrot about 12" away from the knife blade. While this is not ideal, further reduction of this dimension without compromising alignment of the carrot and the cutting board would require a redesign of the hapkit itself. To improve the realism of the cutting board surface, we added a gusseted bracket underneath the knife blade as a hard stop to prevent rotation of the knife below the horizontal.

The insertion hole for the carrot was dimensioned with significant clearance in an attempt to reduce friction, and thus the resistive force felt by the user when moving the carrot. The tradeoff of increasing insertion hole diameter is further reducing the constraint on carrot position relative to the hapkit that determines its position in the rendering. We discovered that it was not difficult to manipulate the carrot in a way such that it missed the hapkit handle. To address this pathology, we explored the following methods of aligning the carrot to the hapkit handle:

Methods for aligning carrot:

1. Joint attaching the carrot to the handle
2. Ceiling to floor constraint on the tip of the carrot
3. Indented channels around the preferred path of the carrot
4. Thicker hapkit handle

The first solution entails directly attaching the carrot to the hapkit handle that determines its position. We considered rigid, hinge, and compliant joints. Since the carrot moves linearly while the hapkit handle rotates, the point of contact between the carrot and the handle changes. As a result, the rigid joint restricts motion, and the compliant joint induces backlash. The hinge joint solves both of these problems; however, it incurs a non-axial translation in the carrot and provides the user a lever arm to transmit a damaging torque on the hapkit handle.

The second and third solutions entail restricting the motion of the carrot. We complemented the ceiling constraint and indented channel solutions with cardboard padding on the inside of the lid to reduce the possibility of the carrot missing the handle when inserted at an angle. These solutions fail because they augment the resistive force felt by the user when moving the carrot. For this reason, we pivoted to maximizing the carrot-handle interface by increasing the thickness of the hapkit handle.

The final solution is a combination of the indented channel and thick handle solutions. To restrict the carrot from going above the handle, we layered 4 additional handles on top of the existing hapkit handle to increase the carrot-handle interface. Due to limited resources, we couldn't employ this same approach to restrict the carrot from going below the handle, so we preserved the indented channel mounted to the bottom of the box. This proved to be an effective compromise.

Finally, we sought to improve the realism of the rendering by adding a removable lid to hide the hapkits from the user's view. We sought to shield the hapkits while maintaining full range of motion on the knife handle. After going to office hours to receive feedback, we found that while the laser cut box had sufficient clearance for a user to comfortably hold and move the knife, this opening gave users a clear view of the fact that there is never a carrot underneath the knife thereby breaking the haptic illusion. Thus, we decided to add small curtains that do not hinder user motion but obscure their view of what the knife is chopping without detracting from the production value of the box.

Current Haptics rending and illusion:

At our last checkpoint, the rendering and illusion of cutting a carrot was at an acceptable level for presentation. For this checkpoint we sought to improve the haptic feedback by trying to achieve the following:

• The knife should not experience any haptic feedback if there is no carrot to cut
• The carrot should not be able to move into the knife when the knife is down on the cutting board.

These goals seem easy but require communication between Arduino boards, which proved to be an incredibly difficult task. Since both boards are writing to serial in order for Processing to read, we had to write code in Processing to write back to the serial of either board. The original goal was to send one board's handle position to the opposite board and vice versa. This is similar to the teleop assignments in homework 5. The errors we encountered could not have been predicted unless we had in-depth knowledge of computer science beforehand.

From Processing, we were able to send floats over the serial as strings. In Arduino, we would read these strings up to the newline and convert them into floating point numbers. However what we noticed was that the floats sent over Processing would get altered. An example would be sending over a value such as 0.0113 and getting a value such as but not limited to: -11403, 6000, 0.0233 or 0.0.0113. We had suspected that strings in C require a hidden null marker \0x8 appended at the end of each string so that C knows the length of the string. More modern languages, such as javascript, don't have this issue. Since Processing was sending floating strings, we believe it did not append this null marker at the end of string. Thus, the Arduino boards were unsure where these strings ended, so when they tried to convert them to floating values, it would produce a value that was entirely unsuspected. Many solutions were tried including appending our own markers and the null marker at the end of the string in Processing, but there was no solution in sight.

Instead we resorted to sending over a string of 0's and 1's to mark different states in either board. For example the knife being up would be a 1 and down as a 0 or the carrot being free to move as a 1 and fixed as a 0. This was sent over Processing as a string "10B". The "B" was used to tell Arduino what to read from the serial. We could then get the boolean character using string.charAt(). From this the board could get rudimentary states of what the other board was doing. We then programmed conditionals to stop haptic feedback in the case that the carrot was not under the knife and haptic walls on the carrot if it were to bump into the knife.

In order to prevent the carrot from sliding "through" the knife when the knife was down, we modeled the knife as a virtual wall acting like a very stiff spring. The equilibrium point of the spring was set to be right at the edge of the knife. We also implemented a visual haptic illusion such that the visual rendering of the carrot does not show the carrot passing through the knife even though it is physically possible to do so. This makes the virtual wall feel like a slightly harder surface and more realistic.