2023-Group 2
Haptic Drum-Hero game
setup
Haptic Drum-Hero
Project team member(s): Ashley Gin, Kaylie Wong, Olivia Tomassetti
Drum Hero is a virtual drumming game (similar to Guitar-Hero) using two drumsticks each attached to a Hapkit, a 1 degree-of-freedom haptic device, and visual graphics displaying the game progression. During the game, the player holds the drumsticks and attempts to accurately strike the virtual drum as notes fall on the drum displayed on a screen. When a drumstick is struck downwards, the Hapkit renders the dynamics of hitting a drumhead. In addition, vibration motors attached to each drumstick vibrate to the beat of the falling notes during the game. For this game, we created several playable levels with different drum patterns for players to replicate.
In traditional rhythm games without haptics, music would be played along with graphics of falling notes, but players who are hard of hearing would experience difficulty playing the game. The purpose of this project is to provide an accessible way for deaf and hard of hearing individuals to play a rhythm game. We hypothesized that adding the vibrations on the beat of the falling notes would improve player performance as compared to without vibrations. We tested our game with six hearing individuals while playing white noise in an attempt to drown-out any sounds coming from the hardware. Although we ran into technical difficulties and inconsistencies during this user study that made the score results unreliable, our qualitative survey results showed that most participants preferred the vibration condition to the no vibration condition. Additionally, through our observations and the participant comments, the user study provided us with many directions for future work.
On this page... (hide)
Video of Drum-Hero in action: Attach:HapticsGameplayMovie.mp4
Introduction
For this project, we wanted to create a fun and useful haptic game that would allow us to strengthen the rendering skills that we learned in class, utilize our controls and dynamics knowledge, and improve our graphics skills. We decided to create a musical game for hard of hearing individuals because we were interested in understanding haptics as a form of sensory enhancement. In creating this game, we took an activity that traditionally required a person’s sense of sound and replaced this requirement with their sense of touch.
Although hardware skills were necessary for this project, our approach also allowed us to focus on improving our game development, drum rendering, and graphics skills. For instance, through this project we performed a study where we varied the dynamics of the drumhead to determine what felt most realistic. We also improved our graphics skills in Processing, a graphical coding platform, and gained experience setting up communication between multiple Hapkit boards and Processing. Through our user study, we were able to observe participants playing the game and make necessary adjustments to our game before our project showcase. The skills we learned and reinforced through this project will likely be relevant if we continue to develop haptic devices in the future.
Background
Previous work has demonstrated the role of vibrotactile feedback in helping individuals who are deaf and hard of hearing feel music rhythm. Researchers developed a device for hard of hearing children that provided visual feedback through LEDs and vibrotactile feedback through vibration motors [1]. Visual and vibrotactile feedback were activated based on intensity of the input sound, and the device was reported to result in positive impressions to users. Another paper demonstrated vibrotactile feedback was successful in helping deaf participants synchronize their bounces to the beat of music [2]. For beginner drummers, researchers developed the Haptic Tutor, a music education tool that provides vibration feedback [3]. Researchers found that participants’ timing accuracy and rhythm precision of drum strokes were improved when haptic bracelets were placed on their wrists and provided haptic vibrational feedback.. These studies motivated the use of vibration feedback in our project to simulate rhythm of music and assist deaf and hard of hearing users with timing accuracy.
Researchers have also developed a haptic drum intended for performers of traditional drums. Their study focused on system modeling and dynamics in order to realistically model the dynamics of a drumstick making contact with a drum [4]. To create a more realistic and physically intuitive experience for the user, researchers noted that the drumstick needs to be forced back upwards after passing through the drum membrane. Therefore, for our project, one requirement for rendering the drum was to provide enough “kick-back” to the user. Our project’s system dynamics were based on the results of this study and revised accordingly to interface with the Hapkit hardware.
Methods
Hardware design and implementation
Our system consists of two Hapkits mounted sideways on wooden stands, each with a drumstick attached at the handle. The handles of the Hapkits were redesigned to accommodate an attachment linkage that interfaced between the Hapkits and drumsticks. This linkage also included an inset to mount vibration motors. Our goal was to make our system as realistic as possible while still utilizing the Hapkit systems provided to us and minimizing system complexity. We decided to use real drumsticks and attempted to replicate a drumming motion to make the user experience more realistic; however, we had to make a tradeoff between closely replicating a drumming motion and limiting complexity in our system.
As shown in the diagram below, a drummer holds the drumstick and the wrist acts as the pivot point. Thus, the drumstick tip travels in an arc. To determine when the user hits the drum in our game, we need to track the position of the drumstick tip. By mounting the drumstick tip to the Hapkit handle, we were able to track the position, but because the Hapkit pivot point is on the opposite side of the drumstick, the path of the Hapkit handle is an arc in the opposite direction of a natural drumming motion. Based on prior research [4] modeling drumstick dynamics, this setup is acceptable for small drumming motions because the rotational angle is so small that the motion can be approximated as a purely vertical path. Additional linkages could be implemented to more closely mimic the path of a drum hit, but we decided to only use one linkage to prevent introducing more friction and weight into the system.
We determined that while playing the game the stands holding the Hapkits needed to resist the force of the user’s drum hit and weight of the system, so L-shaped wooden stands were built. We attached non-skid pads to the bottom of the stands and a weight was placed on top to fix the stands in place.
Side view of one drumstick system. The Hapkit is mounted sideways onto a wooden frame and a drumstick tip is bonded to the Rotational Tab which acts as a pin joint to the end of the Modified Hapkit Handle. In order to provide beat vibrations to the user, an ERM vibration motor is glued into a recess in the Rotational Tab.
Additionally, after encountering issues with the Hapkit boards falling as we played the game, we designed a support piece to secure the Hapkit board in its proper place.
Closeup of the Board Support part
System analysis and control
For our project, we used two Hapkit boards (microcontrollers similar to Arduino Unos) and Processing, a graphical coding platform. The two Hapkits and boards were used to create haptic feedback to the users during the game. One board was attached to the Hapkit with the left drumstick and the other was attached to the Hapkit with the right drumstick. Each Hapkit board tracked the location of the drumstick end (attached to the Hapkit handle) and rendered the feeling of hitting a drum when the handle position was pushed downwards.
Modeling the drumstick to drumhead contact required rendering the drumhead as a stiff wall that provided a realistic “kick-back” to the user and corresponding damping. This was achieved by modeling the drumhead as a mass-spring-damper system (see figure below).
Parameters were tuned (see Study 1 below for more information) and final values used were as follows:
- m = mass of the drum = 5 kg
- b = damping coefficient of drumhead = 1.5 Ns/m
- k_drum = spring stiffness of drumhead = 500 N/m
- k_user = 50 N/m
The force applied back to the user when the drumstick was in contact with the drumhead was proportional to the stiffness of the user hand (modeled as 50 N/m) and the displacement of the drumstick into the drumhead. In order to create a stiffer drum and more realistic sensation, the force applied back to the user was scaled by a factor of 1.8. To calculate the torque of the DC motor required to apply the necessary force back to the user when the drum was hit, the following parameters were used, where torque Tp = force * rh * rp / rs:
- rh (distance from sector center to user hand) = 0.3m
- rs (sector radius) = 0.070m
- rp (pulley radius) = 0.0045m
Software Design
In terms of the graphics and communication, Processing controls the start of the game, sending a start command to each Hapkit board. At the start of each game, “notes,” represented by small circles falling from the top of the screen, start to move down the graphics screen in Processing, moving 2 pixels in the y direction during each loop in the code. When a note reaches the drumhead, Processing sends a command to one of the two Hapkit boards depending on which side of the drum the note landed on. If the Hapkit board receives a command that the note reached the drum in the Processing graphics, then it checks to see if the rendered drum has been hit with the Hapkit. If it has, it sends a correct command back to Processing and the graphics screen flashes green to visually indicate the correct drum hit. The notes on the screen are stored in an ArrayList in Processing so that they can be added regularly and removed once they reach the drum at the bottom of the screen. We also set a threshold of pixel locations that corresponded to a correct hit so that users were more likely to succeed at hitting the drum at the correct time.
In addition to the rendered drum, one of the Hapkit boards also controlled two vibration motors, secured at the attachment linkage on each drum. The motors would vibrate on a beat that corresponded to the falling notes hitting the drums on the Processing screen.
(A) The communication framework between Processing and each of the Hapkit Boards during gameplay
(B) The commands sent between Processing and each Hapkit Board. Note that these commands do not explain the sequence of events during the game'
Software Challenges
Many challenges came up during the software design for our project. For instance, it was difficult to match up the timing of the vibration motors to the visual notes falling on the screen as these were controlled by two different softwares; the vibrations were controlled by the Hapkit boards while the notes approaching the drum were controlled by Processing. The millis() command, which gives the time in milliseconds since the start of the program, was not working accurately on the Hapkit boards so we had to match up the vibrations with a trial and error approach. See the challenges under Checkpoint #2 Section 3 for more details. We also had challenges ensuring accurate communication between the Hapkit boards and Processing as our game required back and forth communication between the two.
Study Designs
Two studies were conducted as part of this project: a dynamics study to select a damping coefficient and a user study to collect player feedback on realism and usefulness of the added beat vibrations.
Study 1 (Controls): We conducted a study to determine the appropriate damping coefficient that would be used to render the drum on the Hapkits. The user (drumstick) position was collected as the drumstick was released from a position of ~0.02 m. Similar to a drumstick being released onto an actual drumhead, we wanted the drumstick to be “kicked-back” and then damped to a stable equilibrium position. Values for the damping coefficient were varied from b = 0 Ns/m to b = 2.5 Ns/m for this experiment.
Study 2 (User): We ran a user study with six participants. During the study, each participant played our thirty-second Drum-Hero game four times. For two of the game rounds, the vibration motors we attached to the Hapkits vibrated as the notes fell onto the drum. For the other two game rounds, the vibration motors were unplugged. For each of the vibration and no vibration conditions, the participants played one round of the “cymbal” level and one round of the “snare” level. The notes on the screen fell twice as frequently in the “snare” level as in the “cymbal” level. During the rounds with the vibration condition, the motors vibrated to the beat of the falling notes corresponding to the “snare” level. The combination of conditions for each round are shown in the table below. We randomized the order of these rounds in an attempt to ensure that any change in score was not only due to participants improving their game strategy.
| Round | Level | Vibrations |
|---|---|---|
| 1 | Snare | No |
| 2 | Cymbal | No |
| 3 | Snare | Yes |
| 4 | Cymbal | Yes |
The script we used to introduce our game can be found in the Appendix. After participants completed all four game rounds with the different levels, we asked them to fill out a survey. The survey included background questions as well as the following questions about their game experience:
- How realistic was the feeling of hitting the drum?
- Which do you prefer? Options: With vibrations, without vibrations, no difference
- How would you describe your experience with playing with the haptic (vibration) feedback vs. without?
- Which was easier to play? Options: Cymbal, Snare, No difference
We hypothesized that participants would prefer playing the games with vibrations as the vibrations should help keep the beat of the falling notes, adding to their musical experience. Additionally, we thought that the “cymbal” level would be easiest as the notes are falling less frequently, giving participants more time to anticipate the notes reaching the drumhead.
Results
1. Dynamics Study Results
Damping coefficient values of b = 1 Ns/m to b = 2 Ns/m provided sufficient kick-back and damping. Values less than 1 Ns/m resulted in underdamping that took too long to reach equilibrium, while values greater than 2 Ns/m resulted in overdamping that did not provide sufficient kick-back to the user. A value of 1.5 Ns/m was selected due to providing the most realistic qualitative feeling of hitting a drum.
Drumstick dropped on drumhead
Drumstick hit while holding drumstick handle (user provides damping)
The plots below show the drumstick tip position over time as the drumstick was dropped on the rendered drumheads with varying damping coefficients.
2. User Study Results
Due to system limitations and differences in how the participants played the game, we decided to neglect the hit accuracy results. Additionally, during the study we noticed that there was a bug in our code that counted some correct hits multiple times, causing inaccuracy in our data. After the study, we were able to fix this bug before the project showcase. Therefore, here we discuss our survey results as well as our observations.
Participant Summary: Through our survey results, we discovered that all six participants in our study have prior experience playing an instrument other than drums and have played rhythm-based games and games which require quick reaction times.
Player Experience: In general, participants felt that playing the game was enjoyable, preferred playing with beat vibrations turned on, and found no difference between the two game levels.
General Comments from Players: Participant comments revealed both strengths and limitations of our system. In general, participants seemed to like the vibrations. One participant mentioned, “With haptics I felt more involved in the game. Without haptics I felt disconnected from the game.” However, participants also commented on the limitations of the system. Writing, “because the sticks were fastened, I didn't feel the freedom of playing drums,” this participant notices the constrained motion of the drumstick when attached to the Hapkit.
Participants also provided helpful feedback to improve our game and our study. For instance, one participant suggested providing players with a tutorial before playing the game, “I was a bit confused if I was supposed to hit when the dot was on the middle line or on the edge of the drum. It might be nice to see a tutorial or make the edge of the drum a lighter color.” These comments are helpful in considering how we can improve our system in the future.
Overall Observations: We noticed that elements of the game that had been intuitive for us as the developers were not intuitive to our participants. For instance, we observed that several participants did not know when to actually hit the drumsticks during the game. We also observed that, even after demonstrating how hard to hit the drum, many participants hit the drumstick harder than necessary, which once resulted in a Hapkit cable coming loose.
During the study, we ran into technical difficulties where we had to reset the Hapkits and restart the game after the Hapkit zero position drifted between rounds. We were unsure how to overcome this limitation but would hope to focus on this in future iterations of the game.
Future Work
In the future, we would start by improving the system based on the results and feedback from our user study. Specifically, we would attempt to make the drumming more realistic by redesigning the current Hapkit handle/drumstick attachment linkage so the drumsticks provide more freedom of movement. Additionally, we would use stronger motors to provide a more realistic drum rendering to the user. We would also attempt to solve the position-drifting problem in the Hapkits so that the virtual drum stays in the same position between levels. This would ensure that users could play our game without resetting the Hapkits between every few rounds.
Many aspects of the graphics could also be improved to create a more intuitive game for users. For instance, we could make it clear where the user should be attempting to hit the drum. Additionally, we could add a training level at the beginning of the game with instructions on how to play. The training level could also include a demonstration of how softly you need to hit the virtual drum to feel the appropriate drum rendering and limit damage to the Hapkit.
After making these improvements, we would test this device with deaf and hard of hearing participants and incorporate their feedback. We hope that participants would enjoy the game and that the vibration feedback would enhance their musical experience. Additionally, we hope that our system and studies could contribute to designing other fun games or useful tools for deaf and hard of hearing individuals.
Acknowledgments
Thank you to Allison, Connor, Dane, and Elizabeth for supporting us throughout the course!
Thank you to our user study participants Rosa, Blake, Rishi, Melissa, Akshita, and Gaby!
Files
Attach:DrumHeroCode
Attach:DrumHeroCADFiles
Attach:DrumHeroBOM
Attach:UserStudyScript.pdf
----
References
[1] H. Florian, A. Mocanu, C. Vlasin, J. Machado, V. Carvalho, F. Soares, A. Astilean, C. Avram, “Deaf people feeling music rhythm by using a sensing and actuating device | Elsevier Enhanced Reader,” Sensors and Actuators A: Physical, Volume 267, 2017, Pages 431-442, ISSN 0924-4247, https://doi.org/10.1016/j.sna.2017.10.034.
[2] P. Tranchant, M. M. Shiell, M. Giordano, A. Nadeau, I. Peretz, and R. J. Zatorre, “Feeling the Beat: Bouncing Synchronization to Vibrotactile Music in Hearing and Early Deaf People,” Frontiers in Neuroscience, vol. 11, 2017, https://www.frontiersin.org/articles/10.3389/fnins.2017.00507
[3] A. Tom , A. Singh , M. Daigle, F. Marandola, M. M. Wanderley, “Haptic Tutor: A Haptics-Based Music Education Tool for Beginners,” International Workshop on Haptic and Audio Interaction Design, 2020, https://hal.science/HAID2020/hal-02901205v1
[4] E. Berdahl, B. Verplank, J. O. Smith, and G. Niemeyer, “A Physically Intuitive Haptic Drumstick,” International Conference on Mathematics and Computing, 2007, https://www.cct.lsu.edu/~eberdahl/Papers/ICMC2007BerdahlEtAl.pdf
Appendix: Project Checkpoints
Checkpoint 1
1. Render virtual wall to simulate drum: We completed the code to simulate a simple virtual drum. We ran into a challenge when attempting to make the drum feel realistic. Currently, the drum is rendered as a virtual wall with stiffness and damping.
Challenges: We tried adding a collision penalty spring force to the virtual wall to create a more “bouncy” drum that would bounce the drumstick upwards after a hit; however, we ran into issues with motor instability. We plan to continue testing and refining the drum dynamics so that it feels more realistic.
Basic Rendering Code:
force = -k(xh - xdrum) - b*dxh
k = 200, b = 5
Desired Rendering Motion (side view): Attach:.DrumRenderingMotion.mov
2. Design attachment linkage for drumstick and Hapkit in CAD: We designed the attachment linkage in CAD. One end attaches to the Hapkit handle and acts as a pin joint, while the other end is attached to the drumstick tip as a fixed joint. Recesses were created in the attachment linkage for vibration motor mounting. We then 3D printed and assembled the first prototype for our attachment linkage. The attachment works well but we will have to widen the recess where the vibration motor be placed. We also plan to add physical limits to the modified Hapkit handle to prevent damage to the system.
Attachment Linkage CAD:
Attachment Linkage Assembled with Hapkit:
Drumstick in Action:
Challenges: We ran into a challenge in limiting the motion of the drumstick. The tip of the drumstick is attached to the handle of the Hapkit which moves in an arc. We decided that we will ask users to make small drum strikes so that the drumming motion will feel more linear.
3. Play with vibration motors and determine duty cycle required to feel vibrations from drumstick: We experimented with different duty cycles to actuate a vibration motor. We decided to secure the vibration motor to the attachment linkage between the drumstick and Hapkit and will place tape on the drumstick at the location where we want the users to hold them. We determined that a 30% duty cycle is the minimum that we will use to provide beat vibrations to the user. We will increase the duty cycle to 50% when a correct beat is hit.
Challenges: Controlling both DC motor for drum rendering and vibration motors – in order for the user to feel the vibration during a correct hit, the vibration motor needs to be turned on for at least 300-500ms. When using the delay() function, the dynamics of the device become unstable due to delays in the loop. We tried implementing interrupts but are having trouble with the logic.
4. Research graphics: We simulated a basic prototype for the graphics. We researched different functions included in Processing that would be helpful for our project. After initial research, we created a screen that has an ellipse representing a drum at the bottom, a line dividing the screen into a left and right side, and circles (representing notes) continuously falling at random times on both sides of the drum. The ArrayList of Objects example on the Processing website was especially helpful when figuring out how to make the notes (which are just ellipses for us) fall.
Graphics Prototype Video: Attach:.GraphicsPrototype.mov
Challenges: We also started mapping out the communication between Processing and the Hapkits. Our plan is to experiment with back and forth communication between one Hapkit and Processing. This Hapkit will then communicate with the other Hapkit. We realized that it will be hard to time everything accurately, especially timing the notes falling in Processing with the vibration beats.
Checkpoint 2
1. Complete initial graphics: We completed the graphics in processing for both the gameplay mode, where “notes” (represented by small circles) will continuously fall on either side of the “drum” (large ellipse on the bottom of the screen), and the menu mode, where players can see their score, select a game level, and play again. We likely will not make many major changes to the graphics besides adding small aesthetic fixes and determining which game levels are sufficient for players.
Challenges: One challenging part of the graphics was setting up reliable communication between the Arduinos and Processing. We started with communication between Processing and just one board. We wanted the arduino code and the Processing code to start at the same time so we could sync the graphics with the timing of the vibration motor connected to the Arduino. To do this, we had Processing write a command to the serial port to tell Arduino to start; however, the same serial port code we were using to tell the arduino that a note landed on the drum was not working to start the game. After lots of debugging, we noticed that adding a delay between when we initialized the serial ports and when we actually sent the command ensured more reliable communication between the Arduino and Processing.
Current Graphics
Game Screen:
Menu Screen:
2. Hardware Prototype: We successfully prototyped our system, with two Hapkits mounted to separate mounting frames made of wood. We also refined the drumstick attachment linkages to house a vibration motor and fit the end of a drumstick.
Challenges: In testing this prototype, we noticed that one of the Hapkits had more friction than the other. We plan to play around more with swapping out parts from our third Hapkit. Another challenge we ran into is that the Hapkit boards are not secure when the Hapkits are mounted sideways. Before Tuesday, we plan to better secure the boards by printing a support piece and mounting it to the Hapkit base. We also hope to clean up the wiring and solder anything as needed.
We are also having trouble with the mounting frames moving while playing, so we may try using the rubber suction feet from the Hapkits or finding a weight to hold them down.
Prototype Setup:
3. Continue working on tuning parameters and graphics: We have been working to try and make the drum hit feel more realistic. One challenge here is that the end of the stick is limited in motion, preventing the hit from feeling very natural. We plan to continue to make improvements with the adjustments. We have also been working on tuning the graphics so that the vibrations from the vibration motors and the falling notes are in sync.
Challenges: The most challenging part for tuning the graphics was ensuring that the notes were falling with the beat of the vibrations. This was a challenge because using millis(), which should give the milliseconds since the program began, did not seem to work accurately in Arduino. We overcame this by selecting a subjectively good beat, determining the beats per minute of the beat, then timing the notes to fall appropriately in Processing.
Current Prototype Video: Attach:CurrentPrototypeVideo.mov
Other Notes
Vibration motors – We initially planned to have vibrations corresponding to every beat and increase the duty cycle when a correct note was played. We quickly realized the motors needed to be turned on for a minimum of 300ms to be felt by the user and using the delay() function would lead to slow downs in our main loop. Instead, we used interrupts to turn on and off the vibration motors, resulting in vibrations every second, but realized turning off the motors when gameplay was complete would require disabling interrupts–which would also disable serial communication which uses interrupts. We are now using the internal timer functions, i.e. millis(), to keep track of the time that the motor was last turned on or off. Due to challenges with timing of the motors on the beat and syncing with the gameplay graphics, we have decided to make increasing the duty cycle when a correct note is played a lower priority; feedback for a correct note is now only visual in the form of a blinking screen in the graphics.
System instability – We noticed that the motors (both DC and vibration) would make high-frequency noises when rendering the drum. After ~15 seconds of gameplay, the DC motor would consistently go unstable and the handle would oscillate. We reverted back to the Assignment 5 mass-spring-damper code with our parameters. This fixed the issue, so we slowly added snippets of our project code into the Assignment 5 solution and have since not noticed any large instabilities.