Gone Fishin'

Fishing simulation device in use on demo day.

Project Team Members: Steven Trinh, Thomas Corcoran, Kareem Dawood, Hanyu Hu

The Gone Fishin' fishing simulator project aims to provide user a realistic fishing experience. The simulator is a one DoF hook that the user controls through the depth of the water to interact with various fish swimming laterally across the screen. It models several different types of fish as a spring-damper system that pull on the hook with a time-decaying sinusoidal function varied by fish type. Two key mechanical assemblies in the system - the reel and the fishing rod handle - simulate the hook raising/lowering and locking of fishing onto the hook, respectively. The haptic system is accompanied with graphics visualization via Processing to help the fisherman see the fish they're trying to hook. The goal of this project was to create a foundation for a more complex mechatronic haptic product that lets users practice their fishing away from the water in a convenient, effective, and inexpensive way. Though elements like fluid dynamics, three DoF hook maneuverability, and rod bending would provide a more realistic experience, they would be too complex to implement. Thus, our project serves as a good starting point.

Figure 1: Team Beside the Gone Fishin' Haptic Product

See a player using the fishing simulation: Attach:GoneFishinDemo.mov

Introduction

The primary motivation behind our fishing simulation project was to provide an immersive and interactive learning experience that bridges the gap between theoretical concepts in haptics and practical, hands-on applications. By creating a tangible and engaging simulation, we aimed to achieve several educational objectives.

Enhanced Understanding of Haptic Feedback: haptic technology is a complex and rapidly evolving field. By developing a fishing simulation, we wanted to give students a real-world application to understand how haptic feedback can be used to simulate realistic physical interactions. This hands-on approach helps demystify abstract concepts and makes the learning process more intuitive.

Integration of Multidisciplinary Skills: The project required the integration of various skills, including mechanical design, electronics, programming, and user experience design. This multidisciplinary approach fosters a holistic educational experience, encouraging students to see how different fields converge in real-world applications.

Promotion of Active Learning: active learning strategies, such as project-based learning, have been shown to improve student engagement and retention of knowledge. By actively involving students in the creation and use of the fishing simulation, we aimed to enhance their critical thinking, problem-solving abilities, and teamwork skills. ''' Simulation of Real-World Scenarios:''' the fishing simulation provides a safe and controlled environment to replicate real-world fishing scenarios. This allows students to experience and understand the dynamics of fishing, such as the resistance of fish and the required skill to catch them, without the need for actual outdoor activities.

Appropriateness of the Haptic Device Approach: the haptic device was an appropriate approach for our project for several reasons:

Realistic Sensory Feedback: the use of haptic feedback through motors and microcontrollers allowed us to simulate the physical sensations associated with fishing, such as the resistance of a fish pulling on the line and the vibrations when a fish nibbles on the bait. This added a layer of realism and immersion that is critical for an effective simulation.

Interactive Learning Experience: haptic devices provide a unique form of interaction that engages multiple senses. By incorporating haptic feedback, the simulation not only relied on visual cues but also provided tactile feedback, making the experience more engaging and memorable for the users.

Immediate and Intuitive Responses: the haptic feedback system offered immediate and intuitive responses to user actions, enhancing the learning experience. Users could feel the direct consequences of their actions, such as adjusting the reel speed or rod movement, which reinforced learning through immediate feedback.

Customizability and Scalability: the haptic system we developed was highly customizable, allowing us to adjust the feedback parameters to simulate different types of fish and fishing conditions. This flexibility made it a versatile educational tool that could be adapted for various learning scenarios and user skill levels. In summary, the motivation for our project was to provide a comprehensive educational experience that enhances understanding of haptic technology and its applications. The haptic device was an appropriate approach as it delivered realistic, interactive, and immediate feedback, making the simulation an effective and engaging learning tool.

Background

The field of haptic feedback has seen significant advancements, particularly in its application to realistic simulations, such as fishing. Several studies have specifically addressed the development of haptic fishing systems, demonstrating the potential of this technology to enhance user experience and training. For instance, Komerska and Ware (2004) conducted a study on haptic linear and pie menus in a 3D fish tank VR environment, which provided insights into how haptic feedback can be effectively integrated into virtual reality settings to enhance user interaction and immersion. This research helped inform the design of our user interface, ensuring that the haptic feedback was intuitive and engaging.

Dobashi et al. (2007) explored a precomputed approach for real-time haptic interaction with fluids, emphasizing the importance of realistic fluid dynamics in haptic simulations. Their work on simulating fluid interactions provided a foundation for creating realistic water resistance and fish movement in our fishing simulator, enhancing the overall authenticity of the experience.

Li Wen et al. (2011) introduced a novel method based on a force-feedback technique for the hydrodynamic investigation of kinematic effects on robotic fish. This research highlighted the role of force feedback in simulating the movement and resistance of fish, which was crucial for developing the haptic feedback mechanisms in our simulator. By understanding the kinematic effects and hydrodynamics, we were able to create a more realistic fishing experience.

Additionally, Kang et al. (2010) developed a haptic fishing robot, which provided realistic feedback to simulate the experience of fishing. This study demonstrated the effectiveness of using robotic systems for haptic simulations and directly influenced our decision to incorporate similar technologies into our fishing simulator. The haptic fishing robot served as a practical example of how to implement realistic force feedback, ensuring that users could feel the resistance and movements of the fish as they interact with the simulator.

These studies collectively underscore the relevance and impact of haptic feedback in simulating realistic fishing experiences, providing a robust foundation for our project.

Methods

Hardware Design and Implementation

Reel

One of the main mechanical assemblies for this project is the reel, which is a belt driven mechanism based on ME327 Hapkit. The reel controls the y position of the hook and gives haptic feedback to the user on fish catching.

Figure 2: CAD of Belt Pulley Fishing Reel

Figure 3: Realized Fishing Reel Subsystem

The belt drive system comprises two pulleys with a 1:3 gear ratio, which amplifies the torque experienced by the user, thereby providing a more realistic simulation. The smaller pulley features 20 teeth, while the larger pulley is equipped with 60 teeth, as detailed in the specifications below. The belt length is 300 mm. Maintaining the appropriate tension in the belt is crucial, which depends on the distance between the centers of the two pulleys.

Figure 4: Specification of Belt Pulleys

To calculate the proper distance between the two pulleys, we used the Timing Belt Drive formula (1) and obtained 106mm is an ideal distance between the centers.

(1)

The small pulley is securely attached to the motor shaft using a press-fit method combined with super glue to ensure rigidity. The larger pulley is press-fitted onto a metal rod, which is subsequently press-fitted into a 5mm bearing. This bearing is then press-fitted into the Hapkit base. To achieve the optimal distance between the two pulleys, we reprinted the base to increase its height.

Figure 5: Exploded View of Reel Subsystem

The usage of the MR sensor is kept as it rotates with the pulley; however, the as the gear ratio is 1:3, the sensor will rotate two times faster(angular velocity) than that of the user input. The MR sensor skipped at high angular velocity so we had to overcome this in the software, for example, limiting the print update rate and using filters.

Handle

The other main mechanical assembly for this project is the handle, which is a 3D printed attachment utilizing a capstan drive mechanism used in the original ME327 Hapkit. The handle allows the user to cast the reel into the ocean, feel the nibbling of a fish, and set the hook on the nibbling fish.

Figure 6: CAD of Handle Subsystem

Figure 7: Realized Handle Subsystem

Enclosure

Housing the reel and handle is an enclosure made of 1/4" thick birch plywood. To keep the enclosure stable and secure during gameplay, rubber suction feet are fastened at each corner of the bottom panel. The back panel includes an inlet for wires powering the motors and Arduinos. The front panel includes a rectangular opening for the handle, while the right panel includes a circular opening for the reel. At the top panel, it is etched with the "Gone Fishin'" sign that includes instructions on how to play the game. On the front, back, left, and right panels, it is etched with the fish that one may catch during the game.

Figure 8: Final Product CAD

Figure 9: Realized Final Product

Code Structure and System Analysis

The overall code structure for the two Arduinos that control the handle and reel haptics systems follow a state diagram structure that flows through the following four states: CastingReq, FreeHook, FishBite, FishHooked.

Figure 10: State Diagram of Overall Simulation

The CastingReq state occurs at the system start and every time a fish is successfully caught by being reeled up to the surface. In this state, the reel is trapped by two virtual walls that prevent reeling from occurring until the haptic fish rod handle has been given a fast, rapid movement. This movement is the event that transitions the system into the FreeHook state.

Within the FreeHook state, the user can rotate the haptic fishing reel in order to both raise and lower the hook in the water. There exists a virtual wall at the top of the rod and the bottom of the ocean that prevent the hook from moving past those points, and any upward hook movement in the water experiences linear damping that is proportional to the hook's upward velocity to simulate movement through water. In this state, the user's goal is to move the hook into the path of one of the moving fish. The intersection of the hook and a bounding box around a fish's mouth causes the transition into the FishBite state.

Once in the FishBite state, the user will experience a vibration in the fishing rod handle, which serves as an indicator to the user to move the fish rod handle in a fast, rapid movement. This handle vibration will continue either until the user has made the rapid handle movement, or until the fish has swam fast the hook. If the user's promptly makes the handle movement, the system will transition into the HookedFish state. However, if the user fails to set the hook into the fish, the simulation moves back into the FreeHook state to find a new fish.

The HookedFish state is responsible for displaying the all of the fish dynamics to the user through the reel. At the depth in the water that the hook intersected with the fish in question, the fish will begin following a time-decaying sinusoidal trajectory relative to the hook's position. This sinusoidal function is distinct to the type of fish, is offset from the hook's position so that the fish is always below the hook, and decays in both wave amplitude and wave speed over time at a constant rate. At any point in time, this will create a displacement between the hook's position and the fish's position, that is used to create a spring force by multiplying by a spring constant for the fish on the hook in the direction of the displacement.

Figure 11: Sketch of Fish Dynamic Behavior

The parameters for the dynamics behavior of each fish type is shown in the following series of arrays:

With xh representing the hook's vertical position, the fish displacement wave function at a given timestep was calculated as:

, where "i" indicates which type of fish (0 to 4) is currently hooked when accessing each array.
Each time the fish's displacement update calculation is called, the linearly decaying of fish amplitude and speed determined by:

The kinematic and dynamic behavior of each fish relative to a fixed hook can be seen in Figures 12 and 13. Figure 12 shows the decaying displacement function that each different type of fish experiences over 10 seconds while in the HookedFish state. One can see that across the fish, the displacement magnitude and wave speed varies dramatically in order to give each one a different feeling to the user when fighting. As time passes, the amplitude and wave speed diminish until eventually the fish is hardly trying to escape at all. Each plot in Figure 13 shows the spring force that each fish is applying to the stationary hook for each of the corresponding fish displacement functions in Figure 12 and each fish type's unique spring constant. The difference in fish feeling is seen more dramatically is these plots, as one can see the rapidly oscillating and large magnitude shark force function compared to the short and more spread-out force function of the angelfish.

Figure 12: Fish Displacement Relative to Stationary Hook

Figure 13: Applied Fish Force on Stationary Hook

Similarly, Figure 14 and 15 show the kinematic and dynamic behavior of each fish relative to a constantly ascending hook over 10 seconds to simulate a user with an infinitely stiff spring constant, with the additional of linear viscous damping through the water. The oscillating fish force can still be seen with the continual rising hook and the opposing movement of the hook away from the fish can be seen to amplify the force the user feels from the fish by about threefold. This fish force amplification and the linear damping force of the hook ascent affects each type of fish identically.

Figure 14: Fish Displacement Relative to Constantly Rising Hook

Figure 15: Applied Fish Force on Constantly Rising Hook

Once the user's hooks the fish and feels the fish pulling against the reel, there are three possible transitions from the state. First, the user can reel the fish all the way to the surface of the water, indicating a successful catch, incrementing the fish counter, respawning the fish at a random y-position at the side of the ocean, and transitioning the system into the CastingReq state. Second, the fish could drag the hook all the way to the bottom of the ocean, causing the fish to get released from the hook and swim away and transitioning the system into the FreeHook state. Lastly, there is a slight random chance, that increases with the fish's time elapsed on the hook, that the fish will release itself from the hook and swim away, transitioning the system into the FreeHook state.

Demonstration / Application

Figure 16: Three Consecutive Catches

Figure 17: Fish Escape at Ocean Floor

Figure 18: Random Chance Fish Escape

Results

In general, we found that the users thoroughly enjoyed the game. Getting the users to understand how to play the game tended to be difficult, so they usually could not catch the fish upon their first attempt. However, after a few attempts and a further instruction, the users were able to catch the fish consistently. Some notable comments as part of the post-game survey include:

• "this was INCREDIBLE -- the wave at the top, the beautiful fish, the fact that the fish can get off the hook randomly."
• "could really tell the difference in forces for the different fish. at one point with the shark, i let go of the handle, and the handle spun which was like actual fishing. Great visuals and the box really pulled off mimicking a fishing rod"
  
• "The vibration after catching a fish is great. It feels like a small fish flopping while on the line."
  
• "i caught an orca!"
  

Below is a plot of the average ratings we received on a scale of 1-5 on three categories in our survey: realism, stability, and noticeability of various vibrations:

Future Work

Our analysis is mainly concentrated on the vertical force (y-axis) acting on the fishing wire; to make it more realistic, we could consider horizontal force (x-axis) as well. One way of adding the horizontal force is to model it based on the angle between the fish and the fishing rod as mentioned in THE FISHING GENE BLOG[5]. The angle will increase as the fish swimming away from the user which results in a large tension in the fishing wire. Since we already have the coordinates of the rod tip and the fish, it's convenient to find the angle between them. After adding the horizontal force, we can add other motor to provide the force in x direction. The force calculation can be written as: Force = Torque /( sine Θ x Effective Rod Length)

We also found the MR sensor performs poorly at a high speed system; as the angular speed goes up, the reading of the sensor becomes unstable and unusable eventually. For a future iteration, we would suggest switching the MR sensor to an encoder for better position tracking.

Acknowledgments

We would like to extend a warm acknowledgement to John Hong, a PRL CA, whose strong acumen in mechanical and product design provided helpful guidance in the development of the project.

Files

Reel CAD: Attach:Reel.step.zip

Handle CAD: Attach:Handle.step.zip

Enclosure CAD: Attach:Enclosure.zip

Hapkit Code: Attach:ArduinoCode.zip

Graphics Code: Attach:ProcessingCode.zip

Matlab Analysis Code: Attach:Fish_Force_Analysis.m Δ

Reel Calibration Data: Attach:Reel_Data.xlsx

Bill of Materials: https://docs.google.com/spreadsheets/d/1STFV844EM3Am2gE3eP3waHyp_dwTrImEk2e7s8xR1wk/edit?usp=sharing

References

[1] R. Komerska and C. Ware, "A study of haptic linear and pie menus in a 3D fish tank VR environment," 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2004. HAPTICS '04. Proceedings., Chicago, IL, USA, 2004, pp. 224-231, doi: 10.1109/HAPTIC.2004.1287200. Available: https://ieeexplore.ieee.org/document/1287200

[2] Y. Dobashi, T. Yamamoto, M. Sato, S. Hasegawa, M. Kato and T. Nishita, "A Precomputed Approach for Real-Time Haptic Interaction with Fluids," in IEEE Computer Graphics and Applications, vol. 27, no. 3, pp. 90-92, May-June 2007, doi: 10.1109/MCG.2007.52. Available: https://ieeexplore.ieee.org/document/4178165

[3] Li Wen, Tianmiao Wang, Guanhao Wu and Jinlan Li, "A novel method based on a force-feedback technique for the hydrodynamic investigation of kinematic effects on robotic fish," 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, 2011, pp. 203-208, doi: 10.1109/ICRA.2011.5979907. Available: https://ieeexplore.ieee.org/document/5979907

[4] J. -J. Kang, Y. -D. Lee and C. -W. Moon, "Development of a Haptic Fishing Robot," 2010 Proceedings of the 5th International Conference on Ubiquitous Information Technologies and Applications, Sanya, China, 2010, pp. 1-4, doi: 10.1109/ICUT.2010.5677879. Available: https://ieeexplore.ieee.org/document/5677879

[5] Paracaddis. (2018, January 18). Trout Torque or Thoughts on Playing Fish. Paracaddis. Retrieved from https://paracaddis.wordpress.com/2018/01/18/trout-torque-or-thoughts-on-playing-fish/

[6] J.-J. Kang, Y.-D. Lee, and C.-W. Moon, “Development of a Haptic Fishing Robot,” in International Conference on Ubiquitous Information Technologies and Applications, IEEE, Dec. 2010. Available: https://ieeexplore.ieee.org/document/5677879

[7] Yong Hae Heo, Seongho Kim, Juwon Um, Gyubin An, and Sang-Youn Kim. 2022. Haptic Interaction Module for VR Fishing Leisure Activity. In Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology (VRST '22). Association for Computing Machinery, New York, NY, USA, Article 61, 1–2. https://doi.org/10.1145/3562939.3565677

[8] Raphael Hover, Gabor Kosa, Gabor Szekely, Matthias Harders. Data-Driven Haptic Rendering—From Viscous Fluids to Viscoelastic Solids. IEEE TRANSACTIONS ON HAPTICS, VOL. 2, NO. 1, Pages 15-27, 2009. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4752819

Appendix: Project Checkpoints

Checkpoint 1

1.) Define the overall goal of the simulation and plan the specific haptic interactions at each stage of the game
The purpose of our project is to provide a haptic fishing game to users, allowing them to feel the sensation of catching and reeling in virtual fish. The product will feature two separate haptic systems that the user can use to hook and reel in fish. The first haptic component is the fishing rod handle, which will vibrate to simulate fish nibbles and bites to the hook. The user will have to pull back on the handle once they feel the fish bites in order to hook the fish and start the fight to reel it to the surface. This haptic handle will be made from a horizontally-mounted capstan drive. The second component is the rod reel, which will allow the user to raise and lower the hook in the water as well as translate the fish forces to the user. We aim to provide users with a realistic virtual fishing experience with the combination of these two haptic components and a graphic display.

2.) Create a state machine of software structure
The overall structure of the simulation is presented by the below state machine. Initially, the simulation will be in the "Free Hook" state, where the user will have a free hook that they must lower into the water from their boat using the DC motor reel. The can raise and lower the hook's y-position by turning the reel CW and CCW, with the goal of getting a fish to bite at the hook. This occurs when a passing fish intersects with the hook, causing the simulation to transition into the "Fish Bite" state.

In the "Fish Bite" state, the user will feel the fish's nibble at the hook through vibrations in the fishing rod's handle. In order to hook the fish, the user must pull back on the handle will enough force to pass the required force threshold of the current fish. If the user is not able to input a force larger than the threshold after a certain amount of time, the fish will leave the hook, transitioning the user back into the "Free Hook" state. However, inputting a force larger enough will transition the simulation into the "Fish Hooked" state.

While in the "Fish Hooked" state, the user's goal is to reel the fish up to the surface without the fish escaping. The only transition out of this state is into the initial "Free Hook" state, which can be done in three ways. Firstly, if the user is able to bring the hook up above the water surface with the fish still on, the fish is considered caught and the transition occurs. Secondly, if the user resists the fish too much, causing the force through the fish line to become larger than the maximum allowable line force, the line will "snap", the fish will escape, and the system will transition to the "Free Hook" state. If the current line force is near the maximum allowable force, the handle will vibrate notifying the user to ease up the line. Lastly, throughout the reeling process, there is a very marginal chance that the fish randomly manages to escape the hook, at which point the system will transition.

Any one of the transitions will reset the system, allowing the user to catch another fish. Additionally, while the user is in either the "Fish Bite" or Fish Hooked" states, any intersections with other fish will be ignored.

3.) Finalize mechanical design concepts and CAD
The fishing simulation product provides the user with two inputs: the hook-set handle represented by the capstan drive hapkit and a fishing rod spool represented by a belt-driven DC motor.

Fishing Coil: First prototype:

This prototype did not change the original dimensions of the HAPKIT; its purpose is to test the idea of belt drive to link the motor and the coiling handle.

Second prototype:

The previous prototype validates the belt drive design so this will be the link mechanism we will be going with. In this prototype, the HAPKIT was reprinted to optimizes the distance between the two gears along with some other small modifications

Next prototype:

With this prototype, the screw will be replaced with a smooth surface rod and a bearing for minimum friction in rotation. The HAPKIT will be reprinted to fit the bearing and damping code will be loaded to the microcontroller to test the rendering experience

Hook-set Handle:

The hook-set handle will be driven by the original capstan drive used by Hapkit, except rotated at 90 degrees to simulate a fishing rod handle grip for the user. In order to keep the Hapkit at this angle, a separate Hapkit stand was created. In this configuration, the Hapkit will be bolted down to this stand, which has its own suction cup feet to prevent movement as the user applies torque to the handle. In order to replicate the feel of grasping a fishing rod, a separate handle was also created. It is important to note that this handle directly interfaces with the current Hapkit handle – it is simply an extension to the original handle. It consists of two mirrored halves that will clamp together the original handle.

With CAD being finalized, the next goal is to realize it via 3D printing.

4.) Start arduino implementation of hook-setting function
Arduino code for reel position tracking and haptic feedback in both the handle and the reel will be completed for checkpoint 2 since the mechanism for the reel must be completed in order to get a feel for our design and how we will go about tracking position.

5.) Develop initial graphics in Processing
Created the initial ocean and fishing boat environment for our simulation which includes a small boat floating on the surface of a low but steadily rocking ocean. A fishing rod is leaned out of the boat with the end of the fishing line currently mapped to y-position of the user's mouse. This mapping will be changed to the position of the handle on the reel component, which will be tracked using the MR sensor and a DC motor. An immediate next step is too add a hook to the end of the line, have that hook move with the reel handle, and add damping on the reeling in and out of the virtual spool. Laterally swimming fish will soon be added that will eventually interact with the hook.

Also, on a different sketch, different fish were imported to the Processing rendering such that they vary in type (and thus size and speed). This will be utilized such that different fish will resist in various haptic feedback to the users experiencing the fishing simulator. The speed of each fish type is controlled, and they appear randomly to add some excitement to the game!

Checkpoint 2

1.) Final hardware at 80% completion
Coiling handle: After all the prototype testing, we found that adding a bearing for rotation reduced the friction significantly. Thus, our driving design is set to be the belt drive that rotates a round a metal rod that is press-fitted into a bearing. However, after testing with the virtual wall code, we found that the 1:1 gear ratio was not able to provide a strong haptic feedback to the user. We then tested a 1:3 gear ratio(20 teeth:60 teeth) to amplify the torque output of the DC motor and turns out well.

Real part:

Hook-Set Handle:

The hookset handle has been printed and assembled as shown in the image. While the look and feel of the handle are satisfactory, the only concern with this mechanical system is that the handle is so large that it may create a strong torque at the handle without any user input. In a future iteration, the handle's thickness and infill can be reduced. The current length is already at a good state and shouldn't be reduced.

Upon creating both the handle and the reel, we realized that it would be beneficial in terms of user experience to create a master stand for the two subsystems in order to create a more cohesive system.

2.) Implement haptic forces into user inputs
We implemented Arduino code to simultaneously track the position of the hook at the end of the fishing line and of each fish, then sent over to Processing to be visualized using the serial port. The hook's y-position in the graphics is mapped to the position of the reel in out mechanical system, so that when the users spins the handle CCW, the hook moves up, and when the user spins the handle CW, the hook moved down on the screen.
We initially struggled with sending five sets of fish coordinates and a hook position, particularly with the frame rate we saw in the graphics and with the hook's position frequently and rapidly jumping around when the reel was turned quickly. We drastically reduced this issue with a combination of reducing the amount of print statements through serial and implementing a moving average filter for the hook position. This greatly smoothed the hook's position on rapid ascent and descent.
Additionally, we incorporated interactions with the hook and fish. When the hook intercepts one of the fish, the fish is now tied to the hook position until the user reels the line up all the way to the rod. At this point, the fish is respawned at a random position.

Gif of User Reeling and Fish Interaction :

The haptic force currently implemented are spring walls at the rod and the bottom of the ocean in order to prevent the user from reeling the hook too high or too low, and damping of the hook moving through the water. The next step will be to add forces associated with fish that are hooked. We plan for these forces will be a combination of several sinusoidal curves that diminish over time to simulate tiring.

We also started defining the different kinematic variables and relations to plan the execution of the simulation. See photos in the following: