Whiplash: Virtual Drum Sticks with Haptic Feedback

Project team members: Chengyi Xing, Jinxin Li, Haotong Han, Teo Ren

This project aims to design a set of haptic devices that can be mounted on any sticks(apple pencil, random wood sticks found outside, etc) to serve as virtual drumsticks. Through the utilization of 9-axe Inertia Measurement Units(IMUs), vibration motors, and teensy 3.2 micro-controllers, we create a realistic drum-hitting experience for the users and a low-cost yet convenient way for them to practice/play drums.

Introduction

Drum players heavily rely on the tactile feedback from drumsticks to perform precise intensity and rhythms. However, setting up real drums can be challenging and time-consuming without proper soundproofing. This often limits practice opportunities, especially for beginners or those living in shared spaces. Virtual drums offer a promising alternative by allowing practice in a controlled volume environment. However, current solutions lack realistic feedback, are costly (~$160), and require additional tracking devices, making them less accessible for many users. Our product, Whiplash, aims to overcome these issues by providing realistic feedback through multiple vibration motors that replicate patterns learned from real drum demonstrations. By integrating haptic feedback technology, Whiplash enhances the learning experience, enabling drummers to feel the nuances of their performance as they would with traditional drumsticks. This realistic tactile feedback helps in developing muscle memory and precise control over intensity and rhythm, crucial for drummers at all levels.

Background

Haptic feedback has been extensively researched in the context of music education, particularly for instruments like drums where tactile sensations play a crucial role in performance. Prior work, such as the Haptic Tutor, has demonstrated the effectiveness of vibrotactile stimuli in improving timing accuracy and precision among drummers. These systems use wearable devices that deliver vibrations to the limbs through vibration motors, helping users develop better coordination and performance. The Haptic Tutor, specifically, employs portable, wireless vibrotactile devices and incorporates inertial measurement units (IMUs) to track drumstick movements and provide synchronized feedback, showing significant improvements in strike accuracy and timing[1]. Additionally, Grindlay and Charles (2007) discuss various haptic devices and their applications in educational tools, again highlighting the feasibility of precise motor control and IMUs for accurate motion tracking to enhance learning experiences[2].

However, such haptic devices mentioned above employ complicated electronic systems that are not only expensive, but also prone to external damage. At the same time, these devices often require the presence of an actual drum set to provide users with drumming education, making their applications to be confined to specific workspaces. As a result, the Whiplash team aims to create a set of drumming devices that allows users to play drums in a fully virtual environment but with high realism. By leveraging more insights from these prior works, including the effectiveness of vibrotactile response for drumming learning as studied by Lee and Choi[3], and the data about fostering of multi-limb rhythm skills through haptic feedback as demonstrated by Holland et al.[4], our team can provide a portable educational tool that supports drummers in developing their skills without the need for extensive setups or expensive equipment.

Methods

Hardware Design and Implementation

The HapStick is an innovative device designed to deliver a highly realistic drumming experience through the integration of three main components: vibration motors, a Teensy 3.2 processor, and a 9-DoF IMU sensor. At the core of the HapStick's functionality are two carefully positioned vibration motors, strategically placed at the tip of the stick to simulate the authentic feel of drum impact. This haptic feedback is crucial in providing users with the tactile sensation of hitting a drum, making the experience immersive and lifelike. The whole drum stick system is shown as below:

In addition to the HapStick, we have developed a feedback device for the user’s shoe that simulates the bass drum through foot stomps. This foot-mounted device incorporates a Teensy processor, an IMU sensor, and two vibration motors, providing haptic feedback to mimic the impact of a bass drum which is shown as the left figure. This integration allows drummers to achieve a comprehensive and realistic drumming experience, engaging both hands and feet. The Teensy 3.2 processor serves as the sophisticated control unit of the HapStick, diligently processing data received from the IMU sensor to manage the vibration motors with precision and accuracy. This processor ensures that the timing and intensity of the vibrations align perfectly with the user’s movements, enhancing the realism of the drumming experience. Central to the HapStick’s capability is the 9-DoF IMU sensor, which captures real-time motion and orientation data. This sensor is essential for precise movement tracking, allowing the HapStick to respond accurately to the drummer’s actions, whether they are fast-paced beats or subtle, intricate rhythms. The HapStick Electrical Overview is shown as the right figure.

All these advanced components are housed in robust 3D-printed mounting cases with Black ABS material. These cases not only ensure the secure placement of each component but also optimize the overall performance of the devices. The 3D printing technique allows for a custom-fit design, providing durability and comfort during use. In summary, the HapStick, along with the foot-mounted feedback device, combines cutting-edge technology and thoughtful engineering to offer a drumming experience that is both realistic and highly responsive, catering to the needs of drummers seeking an authentic and immersive practice tool.

System Analysis and Control

The HapStick's control system is designed to perform a realistic drumming experience through the integration of three core components: vibration motors, Teensy processors, and 9-DoF IMU sensors. The system comprises three distinct pieces: two identical hand-held drumsticks and one foot drum, and each one contributing uniquely to the overall functionality.

Each hand-held drumstick is equipped with a Teensy 3.2 processor that acts as the central control unit, which is used to process data from the IMU sensor and managing the vibration motors. The two vibration motors are strategically positioned at the tip of the stick to simulate the impact of hitting a drum and provide haptic feedback that enhances the realism of the drumming experience. The 9-DoF IMU sensor captures real-time motion and orientation data, which is used to precise track the stick's movements.

For the hand-held drumsticks, when the IMU sensor detects a rotation angle of -40 degrees in the y-axis, the vibration motors and the corresponding drum animation are activated. This threshold acts as a virtual drum face, simulating the action of hitting a drum. Each drumstick can play two different drum sounds, corresponding to two different virtual drum faces. The user can select which drum to play based on the rotation angle of the drumstick's z-axis. The hitting drum face areas are divided into two ranges: 0 to 180 degrees and 180 to 360 degrees.

The left drumstick is configured to simulate hitting two different kinds of cymbals, characterized by longer-lasting vibrations. In contrast, the right drumstick is set to simulate hitting a snare drum and a tom-tom, with shorter, more distinct vibrations. This differentiation is achieved through customized vibration patterns programmed into the control unit.

The foot drum operates similarly, with a trigger range set for angles less than -20 degrees in the y-axis. Upon detection, it activates the vibration motors, sound output, and an animation where the drum logo vibrates in sync with the haptic feedback, adding an immersive visual component to the drumming experience. The control units' block diagram is shown below.

The intensity of the vibration is dynamically adjusted based on the tilting angle of the IMU sensor. The formula for vibration intensity is given by:

where I is the vibration intensity, theta_y is the current angle in the y-axis, theta_threshold is the threshold angle (-40 degrees for the hand-held drumsticks and -20 degrees for the foot drum), and k_d2 is a proportional constant that determines the sensitivity of the vibration intensity to the angle deviation. It is actually a proportional control system (P control). This system continuously monitors the angle of the drumstick and adjusts the motor's output to achieve the desired vibration intensity. The control algorithm can be also expressed as follows:

This P control algorithm ensures that the vibration intensity responds appropriately to the user's movements, providing a realistic and consistent drumming experience.

Demonstration / Application

With the algorithms and device setups described above, we were able to create a set of HapSticks with desirable performance. With a user interface(as shown in the picture below) created using the Pygame package in Python, the device can provide the user with tactile, visual, and audio feedback at the same time. When the drumstick touches one of the virtual drum surfaces, its icon on the laptop will display a vibrating animation and a corresponding sound will be played simultaneously. The user will also feel a vibrating force on their palm.

After some user experiments, we slightly improved the device's wiring and vibration intensity. The final result is shown below:

Results

Parameter setting results

In this study, we aimed to design the stiffness parameters for a simulated haptic feedback system to ensure the feedback is noticeable, mimics real-world drumming, and provides a clear distinction between the two sticks. The stiffness parameters involved are for the left stick (k_l), right stick (k_r), and foot pedal (k_f). The parameters were determined through a within-group user study where participants selected the stiffness values based on their experience.

Each participant was asked to perform the following steps:

- Select a noticeable stiffness for the right stick (k_r), which typically requires less stiffness due to the dominant hand’s natural strength. - Choose a stiffness for the left stick (k_l) ensuring there is a noticeable difference from the right stick. - Determine the stiffness of the foot pedal (k_f) while gripping the drum sticks and using the kick drum. The results of each participant’s selections were recorded, and an average value for each parameter was calculated.

The stiffness values chosen by each participant and their averages are as follows:

The results of the parameter setting study indicate a clear consensus among participants on the stiffness required for the left and right sticks, with average values of 38.75 and 21, respectively. The small variance in these values suggests a shared perception of the optimal stiffness for effective haptic feedback and their difference. In contrast, the foot pedal stiffness showed greater variability, with an average of 30 and a wider range of individual preferences. This indicates that while the stiffness values for the sticks are well-defined and consistent, further investigation or adjustable stiffness options may be needed for the foot pedal to accommodate individual differences in foot strength and technique.

User study results

In our user study aimed at evaluating the perception of our virtual drum sticks, 12 participants were asked three questions: how many drums they could identify, how long it took them to figure out the system, and whether they could correctly feel the difference in stiffness between both hands. Each participant's responses provided insights into the system's effectiveness and user experience. The first question assessed the consistency and reliability of drum identification with the feedback from three modalities, haptic, visual and audio. The second question measured the time required for users to understand the system, and the third evaluated the accuracy of tactile perception regarding stiffness differences.

The first pie chart illustrates the uniformity in users' ability to identify the number of drums within the virtual environment, with all 12 users consistently identifying 5 drums each. This indicates a high level of reliability in drum identification across different users. The second visualization, a histogram, depicts the distribution of time taken by users to figure out the system. Most users required 30 seconds, indicating a generally quick understanding, though a few outliers took longer, with times extending up to 90 seconds, which are also within a very short time. The third pie chart shows the accuracy in perceiving the stiffness difference between both hands. A significant majority, 83.3% of users, correctly felt the difference, suggesting that the system effectively conveyed the tactile differences.

Future Work

Testing the System

To thoroughly evaluate our haptic feedback system, we propose conducting a series of experiments with a larger and more diverse group of participants. These experiments should involve users with varying levels of drumming experience to assess the system's effectiveness across different skill levels. Participants would use the haptic feedback system and provide qualitative feedback on the realism and noticeability of the simulated feedback. Additionally, quantitative measures such as accuracy and timing of drumming actions could be recorded to evaluate performance improvements. A comparison study with a real drum set could also be conducted to measure how closely the simulated feedback matches real-world conditions.

Improvements

Based on initial feedback and test results, several improvements could be made to enhance the system. First, implementing adjustable stiffness settings for the foot pedal and drum sticks could accommodate individual preferences and provide a more customized experience. Enhancing the precision and responsiveness of haptic actuators could improve the fidelity of the feedback. Additionally, refining the software algorithms that control the haptic feedback would ensure smoother and more consistent performance. Finally, developing a user-friendly interface for adjusting settings and monitoring system performance in real-time would greatly enhance user experience.

Applications

Our haptic feedback system has potential applications in several areas. In music education, it can be used as a training tool for drummers, providing realistic practice sessions without the need for a full drum set. Integrating the system with virtual reality (VR) platforms could offer immersive drumming experiences, making virtual performances more realistic. The system could also be adapted for use in physical therapy, helping patients regain motor skills and strength through engaging, music-based exercises. Additionally, incorporating the haptic feedback system into rhythm-based video games could enhance gameplay by providing more tactile and immersive feedback.

Acknowledgments

With many thanks to the TA team and Allison!

Thanks for borrowing IMUs to us, Yimeng!!!

Files

The approximate costs for the whole system: Attach:ApproximateCosts.pdf

The code and CAD models: https://github.com/xcyoloxcy/whiplash

References

List the referenced literature, websites, etc. here.

1. Tom, A., Singh, A., Daigle, M., Marandola, F., & Wanderley, M. M. (2020). Haptic Tutor - A haptics-based music education tool for beginners. HAID 2020 - International Workshop on Haptic and Audio Interaction Design, Montreal / Virtual, Canada. hal-02901205 https://hal.science/HAID2020/hal-02901205v1

2. Grindlay, Graham Charles (2007). MIT. The impact of haptic guidance on musical motor learning. MIT-libraries. https://dspace.mit.edu/handle/1721.1/41562

3. I. Lee and S. Choi, "Vibrotactile guidance for drumming learning: Method and perceptual assessment," 2014 IEEE Haptics Symposium (HAPTICS), Houston, TX, USA, 2014, pp. 147-152, doi: 10.1109/HAPTICS.2014.6775447 https://ieeexplore.ieee.org/abstract/document/6775447

4. Holland, Simon; Bouwer, Anders J.; Dalgleish, Mat and Hurtig, Topi M. (2010). Feeling the beat where it counts: fostering multi-limb rhythm skills with the haptic drum kit. In: Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction, 25-27 Jan 2010, Boston Cambridge Mass, USA. https://www.researchgate.net/publication/221308797_Feeling_the_beat_where_it_counts_Fostering_multi-limb_rhythm_skills_with_the_haptic_drum_kit

Appendix: Project Checkpoints

Checkpoint 1

In our project, we approached the development from both hardware and software perspectives to ensure a comprehensive and integrated solution. On the hardware side, we successfully designed the CAD model for the haptic device and completed the first version of the prototype. The design phase involved creating a detailed and functional model that could be easily mounted onto various sticks, such as an Apple Pencil or a wooden stick. The prototype included mounting features for the IMU sensor and the vibration motor at the tip, which are essential for simulating the drum-hitting experience.

On the software side, we achieved a significant milestone by implementing object detection using the Segment Anything Model (SAM). This allowed us to accurately identify and track the wooden stick within the camera’s field of view. The SAM algorithm proved to be highly effective in distinguishing the drum faces from the background, which is crucial for real-time motion tracking and interaction.

While we have made good progress, there are still ongoing tasks and challenges that need to be addressed. One of the main challenges we faced was the calibration of the IMU sensor. The calibration process is critical to ensure accurate motion tracking, but it has been more complex than anticipated due to the drifting. We are currently in the process of refining the calibration method to achieve our goal.

Another challenge was integrating the software and hardware components. Ensuring that the real-time data from the IMU sensor and the visual data from the camera synchronize correctly requires meticulous coding and testing. However, we are optimistic that with continued effort, we can overcome these difficulties.

At this point, we don't have too many changes to our original plans. Initially, we chose two collinear vibrating motors for our tactile feedback, but we decided to use two motors placed in a way to form an equilateral triangle with the center of the rod to amplify the vibration intensity. This adjustment aims to enhance the tactile feedback experience.

Checkpoint 2

In the past week, we successfully configured the MPU6050 IMU through Teensy 3.2 and performed calibrations on the IMU's real-time linear and angular accelerations. Then, we integrated the electronics system with the previous mechanical structure to form a prototyping drum stick, which is shown in the picture below:

This marks another vital milestone to our project as we were able to achieve basic desired functions of a drumstick: when detecting a combined linear and angular acceleration above a preset threshold, the vibration motors will react correspondingly in different conditions(e.g different gongs and drums)to provide feedback for the user. At the same time, we utilized a laptop speaker to play sounds to improve the user's experience.

However, we did face some challenges regarding capturing the drumstick through camera. Our objective is to create a 3-D virtual world that has multiple drum faces for the user to play with. While we were able create a 2-D flat world from a laptop's webcam, we could not track the drumstick's depth and add an extra dimension.

To solve this, we plan to utilize the IMU's linear acceleration to calculate the drumstick's displacement in-to-the-page. In the following days, we will create a visualization of the virtual drum for the user to see the drum-hitting process. We will then integrate all components together to make further improvements on users' experience. Eventually, depending on how much time we have left, we will duplicate the current system to create the authentic drum-playing experience with 2 sticks.