Haptic Fruit Ninja

Project team member(s): Joshua Lin, Yichi Ma, Karim Rishani, Catherine Xiang

Introduction

A haptic device interacts with both reality and virtual environments. Haptic feedback can be integrated into various applications, including gaming, surgical operations, and education. As the e-sports and virtual/argumented reality becomes more popular, there is a demand of combining haptic in the user experiences. In this project, we developed a pantograph device to enhance the Fruit Ninja gaming experience by utilizing force feedback and precise positioning with motors and MR sensors. Our project takes a popular game, Fruit Ninja, from a purely virtual game on a tablet or phone to a physical world interaction through manual manipulation of the pantograph handle. The hardware and the software integration explore and expand haptic use cases in the gaming, health, or education industries.

Background

We chose to use a 2-DoF pantograph system to implement an improved gaming experience via translation force feedback and haptic vibrations to better immerse the user in the game. The device will use force feedback and forward kinematics to create realistic haptic feedback.

The paper by G. Campion, Qi Wang, and V. Hayward introduces the Pantograph Mk-II, a haptic instrument aimed at providing realistic virtual interactions based on a two degree-of-freedom parallel mechanism. The following are the derived equations of motion for the 2-DoF pantographing consisting of a 5 bar planar linkage. From these equations, we can calculate the Jacobian can be derived and apply it to our own project’s force feedback. Implications for our project include leveraging this efficient solver for our haptic device design, potentially enhancing computational efficiency and real-time performance. The paper's experimental results serve as a benchmark for evaluating our prototype's performance.

In the paper by M. Friedel, E. Sharlin, and R. Suzuki, a 3D Pantograph mechanism is explored to provide the user with force feedback in a virtual environment. Their “HapticLever” uses a 4-bar linkage characteristic of all pantographs except with one set of arms being much shorter than the other as shown in Figure 1. This “small-scale arm” is used to provide physical force feedback to the user who interacts with the end of the large pantograph arm. Friedel et al used scale models of various surfaces rigidly mounted underneath the small-scale arm to simulate larger textures/surfaces felt at the user’s interface node as seen in Figure 2. The proposed benefits of this system include greater strength than direct actuation counterparts due to its passive nature, energy efficiency due to its purely mechanical nature, and transparency in free space as there is no rigid connection between the constraint node and the sample surface.

Methods

Hardware Design and Implementation

Fruit Ninja game is known to user that the blade is moving in a plane. Thus, a two degree of freedom device is needed to simulate the game. As a result, we chose a pantograph for user input hardware design. Under the time and budget constraints, we utilized the pantograph design originally from the GraphKit. In this four bar linkage system, the upper linkage is controlled by the left motor, while the lower linkage is controlled by the right motor. Two motor positions determine the user handle in x and y space. We made several modifications in this project to expand the workspace of the end effector.

Our design involves in create two 180-degree sector, which consists a larger angle of linkage swing. This results in a larger workspace in y directions, covering all gaming space shown on the screen. Our final sector design included 10 degree increments for easier calibration.

The game handle with a circular cross section, imitating the sword handle, is attached at the tip of the two linkage joints. To avoid tipping over of the device when the pantograph extends to the maximum workspace, we installed a blade shaped support below the game handle. A roller will be installed on the tip of the support to minimize the friction between the support and the support.

System Analysis and Control

For our haptic interface, we utilized a 4-bar linkage, commonly referred to as a pantograph. This pantograph design allows for a 2-degree of freedom workspace. The initial step in analyzing this system was determining how to obtain the x,y coordinates of the handle on the pantograph. Below is a diagram of our pantograph, highlighting the relevant unknown angles and constant lengths.

θ1 and θ4​ are known since they are determined by the rotation of the motors, which in turn rotate the two bottom arms of the pantograph. The lengths of the pantograph arms are also known; however, θ2​ is unknown. θ2 represents the angle between the horizontal and arm B. To find the x and y positions of P3​ (denoted as xh and yh​), we first need to calculate the vector from P1​ to P3​, then take the dot product with the horizontal and vertical unit vectors nx⃗ and ny⃗​.

Given the known angles θ1and θ4​:

To determine θ2​, we split it into two angles, α and β:

First, we calculate angle α using the law of cosines. Lh is the length between P2​ and P4​, which divides our system into two isosceles triangles. From the law of cosines, we get:

To find β, we get the unit vector from P2​ to P4​ (denoted as hx⃗​​) and take the dot product with the horizontal unit vector nx⃗​​, which gives us cos(β):

Using the equations above, we solve for the handle coordinates.

To further analyze the dynamics of our haptic interface, we need to determine the derivatives of the handle coordinates xh​ and yh​. These derivatives are essential for understanding the velocity of the handle in the workspace and will be necessary to calculate any damping force we want to add to the handle. Given that θ_dot1​ and θ_dot4​ are known, we will focus on explicitly expressing θ_dot2​.

To find the derivatives of xh and yh, we apply the chain rule:

Now, we need to express θ_dot2​. From the previous analysis, we have Θ2​ = α − β where:

Taking the derivative of θ2​:

We calculate the derivative of ​α and β using the chain rule:

Due to the complexity of the derivative of β we keep the second part of the derivative simplified.

Finally, substituting θ_dot2​ back into our derivatives for xh​ and yh​:

This provides us with the necessary expressions for the Jacobian and allows us to understand the velocity components of the handle in the workspace.

Understanding the velocity of the handle was necessary for the force feedback since we wanted to add damping to the handle when necessary. The damping equations are:

Fx = - b * xh_dot

Fy = - b * yh_dot

To implement this in Arduino, we used the current and previous position coordinates to calculate xh_dot and yh_dot. We then filtered these values to reduce the noise associated with the measurements.

Modeling Game Dynamics

A collision was defined as when the pantograph end effector (user's sword) overlapped with the constant radius of a Fruit Ninja object (fruit or bomb). The force feedback would then apply damping on the user's sword, which is a function of b (a defined damping constant) and the sword's velocity. Thus, a faster slice will result in a larger resistance and an increased sense of difficulty in cutting the fruit. The intended Fx and Fy (intended applied force in the x and y directions) was then translated into torques in the respective motors. Later, we found that small velocity changes would overwhelm the motor as the data is uploaded in every Arduino loop, so we implemented a filter that did only updated the force feedback if the velocity was greater than a defined value to make the damping feel smoother.

We tuned the damping constant between slicing a fruit to be low enough to feel a resistance or "fruit texture", but not inhibit motion based on our experiences with the Fruit Ninja app game. To create a realistic haptic feedback for hitting a bomb (explosion), we tuned the damping constant for the bomb to a higher value, enough to make the two motors oscillate and create large vibrations against the pantograph end effect (user's sword).

Graphic/Sound Interface

For the graphic and sound interface, we used images of common fruits similar to those found in the Fruit Ninja video game, along with typical slicing sounds. The fruit images followed the x and y positions calculated by the Arduino for their paths. These coordinates were printed in Arduino and then fed to Processing. The slicing sounds were triggered when a collision occurred. In the Arduino code, a print statement changed from 0 to 1 when a collision was detected. This print statement was used to activate the sound in the Processing environment. To have a matching collision between the graphic of the fruit, and when one was technically detected in Arduino, we had to stick to round fruits. This was because the collision detection was based on a radius from the x, y coordinate of the object, so the fruit had to have a circular frame. Furthermore, on top of a slicing sound when collisions happened, we also changed the graphic of the fruit (sliced version of fruit) to give visual feedback of the collision.

Demonstration / Application

Short example of Fruit Ninja Haptic at the Haptics Open House:

See 'Files' section below to view a full video of one play of the game (start -> bomb) with sound.

There are numerous applications for the methods, designs, graphics, and control logic in this device.

As haptic games become more increasingly widespread with the improvement of both realistic haptic feedback due to continued research int he field and the increased popularity of VR, this can serve as one game of many to shift from a 2D planar surface of a phone or tablet device into a 2 DOF pantograph that provides more realistic game feedback that allows the user to feel as if they are physically immersed. The game elicited smiles, gasps, and cheers among the individuals who played it (and waited in a line to play it) at the Haptics Open House. This makes our generation reminisce in the games of our childhood, which are now improved with haptics. At the end of the day, haptics can serve as a way to make people feel more connected.

Results

Describe the results, which may include qualitative responses from users at the open house.

A total of 22 responses are collected in this study. "Realistic" and "vibrations" are the words that appear the most in the "How do you felt when you cut the fruit" answer sections. "Fun," "nice," and "smooth" are the second frequent words appear in this section. The user responses showcases that the force feedbacks during the game play are obvious and realistic. Other comments mentioned that the force feedback from hitting the bombs is much greater than hitting the fruits, which is expected.

According to the survey, 90.5% of respondents are interested in buying this game, while 9.5% say they will not. Among those not interested in purchasing the game, the primary reasons are the lack of information about the game's price and the fact that some individuals do not play games at all. In summary, the haptic integration of Fruit Ninja game provides a great user experiences.

Participants also provided some helpful feedbacks to the hardware and user interface design. For instance, one participant mention that "mechanism seems to have some play when end effector is far from the base." Another participant also comments on the limitation of the game interfaces, "future work is rendering the fruits rotating in air."

We observed that the current hardware setup is not the most convenient. The handle is placed too far from the user's standing point, causing the user's arm to easily interfere with the exposed jumper wires of the haptic hardware. Additionally, during the study, we encountered a technical issue where the handle became loose after extended use. However, this should be an easy fix.

Future Work

Hardware Changes

While our pantograph system survived demo day, the top handle of our end effector broke off due to repeated torquing throughout the gaming sessions. To improve physical durability, we’d like to develop a more robust threaded attachment for both end effector halves instead of the current press fit method. Another design change that would reduce movement friction and reduce damaging torques would be to add a rolling ball point bottom end effector that would provide smooth motion and extra support for the currently cantilevered pantograph system.

Although our workspace is already much improved over the standard graphkit, longer links would enable us to further expand our workspace and ensure that the pantograph can reach all corners of the rectangular play area. To further improve the player experience, we would like to upgrade our tracking system to use encoded motors as the current magnetoresistive system tends to lose tracking and must be reset with the reset button when the player moves too fast and encounters violent FFB.

As quality of life upgrades, we would like to upgrade the current bushing system to use ball bearings for an even smoother user experience and sort out the current birds nest of wiring that may impede user motion. Additionally, a counterweight system in the base would be convenient in case the user is playing on surfaces that the suction cup feet are not able to successfully seal against.

Game Design

To make gameplay more satisfying, we would like to implement more realistic force feedback based on the fruit sliced. We plan to conduct real life testing on fruits to determine stiffness differences in the skin, rind, and flesh of fruit with the ultimate goal of creating accurate textures for individual fruit in the game. In order to make the game more competitive, we would like to enable multiple fruits being on screen at once and track if the user slices them in one continuous slice for a “combo,” earning them bonus points. To further deepen gameplay, we would also like to include special effects fruit that temporarily alter the game state e.g. slowing down time, creating a fruit frenzy, or multiplying points for a certain amount of time. Finally, in order to limit play sessions, we would like to implement a timer function in our code that aligns with the current soundtrack.

Acknowledgments

Thank you to Professor Allison Okamura, CA Dani, and other peers for your feedback (and not the force kind). We appreciate it!

Files

Full Gameplay Video

Code Zip for Leader, Follower, Graphics, and Previous Work

CAD File Zip Folder

Bill of Materials

References

[1] G. Campion, Qi Wang and V. Hayward. Efficient Pantograph Kinematic Solver: The Pantograph Mk-II: A Haptic Instrument.” 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2005, pp. 193-198, doi: 10.1109/IROS.2005.1545066. https://ieeexplore.ieee.org/document/1545066

[2] Marcus Friedel, Ehud Sharlin, and Ryo Suzuki. 2022. HapticLever: Kinematic Force Feedback using a 3D Pantograph. In Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 47, 1–4. https://doi.org/10.1145/3526114.3558736

[3] Nianlong Li, Han-Jong Kim, LuYao Shen, Feng Tian, Teng Han, Xing-Dong Yang, and Tek-Jin Nam. 2020. HapLinkage: Prototyping Haptic Proxies for Virtual Hand Tools Using Linkage Mechanism. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, Virtual Event USA, 1261–1274. https://doi.org/10.1145/3379337.3415812

---

Appendix: Project Checkpoints

Checkpoint 1

Our goals for this first checkpoint were: (1) to understand the kinematics of the GraphKit, (2) make significant progress on the mechanical design of the system, and (3) begin the software development for the interface between controller and program.

(1) Kinematics

To model the kinematics of the HapKit, we created the following diagram:

x and y define the location of P3 relative to P1, or the origin point. Then, given our knowns and unknowns, we derived the following equations and will ultimately use x’ and y’ in our Jacobian that we put into the Arduino code:

Our next step is integrating this Jacobian into Arduino.

(2) Mechanical Design This week, our group tested the modified GraphKit linkage based on a previous ME327 project (V1) and thought of further improvements to the system to increase the GraphKit’s workspace and ease of assembly. We will be modeling and prototyping these improvements in the coming days to test with We successfully 3D printed the V1 linkages and the sectors this week and conducted initial testing with the pantograph. Our joint connections are currently implemented with shoulder screws, bushings, and washers. We are currently debating if we will use ball bearings going forward or if the gains would be minimal compared to our current bushing setup.

We have completed the first task to fabricate the Graphkit pantograph. For the future step, We need to identify the system's workspace to ensure it approximates a rectangular shape. Currently, the sector motion is limited to 90 degrees. We plan to modify the sectors to extend to 180 degrees to fully maximize the workspace. We also aim to design the linkage connections, and motor and microcontroller mounts accordingly to account for the increased range of motion. We want to limit the motion of the pantograph within our game space. Since our actual game space is a rectangle, we plan to impose a virtual wall for this rectangular space by implementing a stiff spring when the pantograph end approaches the border of the game space.

(3) Software Our group plans to use a modified version of the HapKit Arduino code that has been developed throughout the quarter to sense the position of the HapKit lever. Our code currently tracks the position of each individual HapKit well and our next step is to connect both HapKit boards/motors to one Arduino program.

Our group has also started developing the Processing GUI that will display fruits for the user to slice. To imitate the “tossing” behavior, we set the fruits to have initial x and y velocity coming from the bottom of the screen. “Gravity” was added by subtracting the y velocity over time. Fruits will be tossed once all of them have disappeared off the screen. One to three fruits will spawn next with a random initial velocity. Here is the GIF example:

Checkpoint 2

Here you will write a few paragraphs about what you accomplished in the project so far. Include the checkpoint goals and describe which goals were met (and how), which were not (what were the challenges?), and any change of plans for the project based on what you learned. Include images and/or drawings where appropriate.

Example Video: https://www.youtube.com/watch?v=i_aLBql4Ufo

(1) Hardware

Our group has successfully fabricated the pantograph holder by using laser cutting the acrylic board. We have also assembled the pantograph onto holder by using 1/4 in D shaft, washers, and shaft collar. The pantograph joints have been modified since checkpoint 1. The joints of the pantograph now consists the shoulder screw and the lock nuts. The washers are placed in between the two linkages and the locknuts to eliminate the friction.

For the installation of the pantograph, the left linkage is mounted on the top of the center beam of the holder structure while the right is positioned below the entering beam to avoid interferences. When the pantograph is fully compressed, the linkage will fit into the slots of the holder. The minimum angle the linkage can extend without hitting the pantograph holder is around 25 degrees.

Right now, our pantograph sectors are only 90 degrees but our next step will be to test new 180 degree sectors that will afford us a much wider workspace. Our group has made a preliminary plot of our workspace with the current 90 degree sectors and will create a new workspace graph with the 180 degree sectors. We also plan to fabricate a handle on the tip of the pantograph by using 3D printing. After the fabrication of all the mechanical components, we will install the motors and HapKits with MR sensors onto the central beam of the pantograph holder to test the software.

(2) Software

Since Checkpoint 1, we implemented Arduino code for the simultaneous position tracking of the two pantograph arms via a primary board and secondary board connected through the serial port. We have tested the code with LEDs to verify that both positions are being tracked and are currently working on integrating this code into the rest of the project.

We have also made significant improvements to the Arduino/Processing code responsible for tracking both the watermelon and the handle. Now, both tracking tasks are handled by the Arduino, which prints and registers the x, y coordinates of both objects to Processing for display.

Below is a screenshot of the Serial print. The first set of coordinates circled in red represents the coordinates of the watermelon. The next set corresponds to the coordinates of the handle, and the final value indicates whether there is a collision (1) or not (0). Collision detection is performed by checking if the handle coordinates are within a certain radius of the watermelon coordinates. While the kinematics for the position of the handle have been developed, the angles have been set to constant values. This is because we currently cannot use the pantograph to vary the two known angles, which are then used to solve for the handle coordinates.

While the kinematics of the position of the handle have been done, it has been set constant since we can’t yet vary the two known angles with the pantograph.

Below are the rendered graphics. The handle is represented by the magenta circle. It remains stationary in this case because we are keeping the pantograph's angles constant. As shown, when the watermelon enters the range of the handle, a collision occurs. This collision is then graphically rendered.

Note: the watermelons have been oversized to increase the frequency of a random collision

Goals and Future Work

After installing our new sectors, we plan to implement a bounding box within the workspace that will reflect the square “dojo” window shown to the user. Additional force feedback that we need to implement includes the forces felt when chopping a fruit (dependent on the fruit’s velocity vector and the user’s velocity vector), implementing haptic bombs, and a score tracker to count the number of fruits sliced. A longer term goal for us is implementing a variety of fruits with different sizes and force feedback values.