Bio-feedback Heart Beat

Aziz Nawrozie, Anna von Wendorff, Pajnucci Vue

Introduction

HeartBeat is a haptic feedback device that allows users to feel their heartbeat in their hand. The user attaches a sensor to their wrist or finger that tracks their heart beat. This signal is then echoed into a realistic, two chamber silicone heart which the user can hold and feel expanding and contracting in the same rhythm as their own heart. This project is inspired by biofeedback and the positive effects it has been shown to evoke. This project can be used in both the medical space (e.g. to improve patient reactions to mindful meditation exercises or anxiety therapy) and in education (e.g. to teach about heart anatomy)

We investigate two things: 1) Bio-feedback impact: whether a user's’ heartbeat changes as their simulated heartbeat (the device they are holding) increases or decreases, and 2) how external stimuli (different kinds of music) changes the heartbeat.

Design Iteration 1 (left) and Design Iteration 2 (right) and Final Design (below)

Attach:IMG_7386.MOV

Background

There is much research into biofeedback haptic devices, as well as multiple consumer goods on the market which are in this space. Many of these devices use vibration or visual feedback to communicate biological triggers like breathing rate, posture, or weight bearing to users.

Fu et al (2014) use real-time haptic (vibratory/vibrotactile) biofeedback to enable patients to better comply with weight bearing instructions. Many patients are instructed to limit weight on their lower extremities following orthopedic trauma and surgery. The authors created a device which vibrated if they exceeded the acceptable range, and tested this system against conventional physical therapy methods like bathroom scale training and verbal instruction. The study found that using haptic device decreases the amount of weight placed on the limbs by half. The amount of weight placed on their lower extremities as a percentage of body weight was 40.2±19.3% for the verbal instruction group, 32.5±16.9% for the bathroom scale group, and 14.5±6.3% for the haptic biofeedback group. This shows how promising vibrotactile haptic devices are in the therapeutic space.

Giggins, Persson, and Caulfield (2013) survey additional (non-vibrational) ways in which haptics have been used to aid in rehabilitation. The cardiovascular biofeedback section in particular notes the effect biofeedback of heart rate has on the user’s actions, citing that for recovering patients, being aware of their heart rate helped keep it lower as they exercised (a positive outcome). It also discusses different studies conducted that focus on displaying the heart rate variability, citing that displaying this to patients has helped mitigate negative effects of heart rate fluctuation in a wide range of diseases. These are not simply improvements to heart rate stability alone but also involve improvements in depression and reductions of stress.

Janidarmian, et all (2016) investigate whether breathing therapy can be made more effective through haptic (vibrational) biofeedback. Breathing therapy helps patients adjust their respiration rate and volume, and has been shown to be effective in lessening the symptoms of multiple chronic diseases, and improving emotional wellbeing. Unlike this study, however, we do not plan on using vibrotactile feedback as the primary feedback method. The authors created a haptic device which consisted of 2 parts: a belt which vibrated and an additional device which was on the abdominal region to register signals. The impact of this device was measured by an experiment of 10 volunteers each conducting 5 breathing exercises with and without the haptic device. The study found a 40% improvement in users’ performance in breathing therapy when the device was used. One drawback of this paper, however, is that the device only helps users follow directions more accurately; it does not explicitly relate to patients how they can improve their performance.

Almost every action done by a human effects their heartbeat: sleeping, exercising, speaking, listening to music, and even feeling certain emotions have been found to increase beats per minute. Koelsch and Jaencke (2015) conducted a study and literature review showing that music can powerfully evoke and modulate changes in heart activity, blood pressure, and breathing. We decided to use this stimuli in order to induce changes to a user’s heartbeat so that they could see their heartbeat changing, thus allowing them to observe the dynamic aspect of our system in a short “demo” time-frame.

One of the first decisions we had to make when building the device was how to measure the heartbeat of the user. We conducted some searches on types of sensors in haptic devices that measure beats per minute and amplitude of a user’s heart. Zhu, Li, and Pan (2018) present some of the most recent research in pressure sensors. Their design for an epidermal iontronic sensing (EIS) device is promising because it eases data collection by attaching directly to the skin and because of its small, lightweight design. The sensor also produces data with less noise than other pressure sensors. Because of the innovative approach to having human skin as the electrical return and the ground of the sensing device, the device is can accurately measure blood pressure pulsations. In a future iteration of our project, we could incorporate the EIS device in creating a more discrete way in how we sense the user’s heartbeat.

Methods

Provide a detailed description of your project, such that another student from the class could generally re-create your project/experiment from the report if necessary. (You don't need to document every screw, but the design should be clear.) Add images and videos as needed to support the description. You can refer to downloadable drawings and code in the "Files" section (later). You should divide this section into subsections, which can vary depending on your particular project. Here is an example set of subsections:

Hardware design and implementation

For our heartbeat sensor, we decided to use the Sparkfun MAX30105 breakout board, which houses an optical sensor that can use red, green, and IR LEDs to measure the changing in reflectivity of a surface. If something is kept close and still enough, it can optically measure a heartbeat. This breakout board also has an arduino library to handle initialization, as well as documentation on how to use it to measure a heartbeat.

Using our hapkit microcontroller, we connected the breakout board and used the libraries to set up a sketch that would plot reflectivity results using only the IR led (which performed better than red and green in tests). We played around with different ways to record a measurement, and ultimately ended up with having the user rest their hand on a table and strapping their index finger using velcro on top of the sensor. By having the velcro strap holding the finger in place, this reduced variations in signals that came with a user applying their own force. To help stabilize the data further, our code includes a function that takes an average of wave values and subtracts it, much like how a high pass filter circuit would remove the DC components of a signal. The result is a PPG signal that had the correct waveform with a measurable, stable amplitude and frequency.

To help facilitate an ergonomic experience, we created a housing for the sensor circuitry that included a contour to match a hand. The quality of the print was lower than we hoped, and so we decided to cover the surface of the box with fabric, which we hoped still contained some of the contour benefits while masking the print layers.

For the physical silicone heart model, we silicone cast the heart to have a ~0.75" thickness using a 3D printed mold with an updated undersized core. The thicker wall made the model feel heavier and more realistic, and any actuation will appear over a larger area that in a thinner membrane.

We developed a motor mount that houses the motor. The mount has a flange that sits at the mouth of the cavity inside of the cast silicone heart. We chose to use a 170 rpm motor with integrated planetary gear train because of its high torque and because our mechanism, a rack and pinion, involved very few motor revolutions. Another reason we chose to use this motor is because it has a d-shaft which prevents the pinion gear from slipping. The rack and pinion mechanism was lasercut and dropped in/press fit into our assembly. The racks include stops that to prevent them from overtravel.

Our first iteration of the rack was cut too short, which meant that most of the rack’s travel was spent disengaged from the heart wall, and it would only create a light tapping upon contact. This provided small vibratory feedback, but didn’t show much deformation of the heart. We addressed this by making a longer rack that would be engaged with the heart walls for all of its travel. We also took this opportunity to situate the motor deeper inside of the heart cavity so that that the actuation was closer to the ventricles, and added a large shape to the end of the racks to make the appear less like a point-force. It was difficult to find exactly how far the heart expands from this “default” state, which helps us determine the actual size of the rack.

We tested multiple different actuation patterns to find the one that was most compelling. Originally we wanted the heart to expand and contract twice, once for the “lub” and once for the “dub” of a heartbeat. In testing, we found that while this was more realistic of a heart motion, the auditory noise from the motor retracting gave the impression that the heart was “beating” 4 times instead of twice. In our final actuation, the rack moves forward for the lub, and backwards for the dub.

System analysis and control

The system is dynamic as it can react to the users’ heartbeat changes. When first planning out the software, we looked at samples of heartbeat data we found online. A healthy heart contracts (empties blood) and relaxes (refills blood) 60-80 times each minute. With each heartbeat, the heart pumps blood from the left and right ventricle. We created a small set of points that created an example normal heart beat plot, and scaled those values for use as input data in our project:

We first investigated how a heart expands and contracts, mostly by looking at animations and actual anatomy example videos. It was important to us to make the “lub-dub” feeling of the heart as realistic as possible. A study by Gammill et al in 1970 looked at the roentgenographic changes in heart size from systole to diastole showed that size changes in the width of the heart varied from 10% to 40% for the majority of the patients in the study. Because the heart width is approximately 4 inches, we wanted a max expansion of between 0.4 and 1.6 inches. For ease in calculation, we made both racks move by 1 inch, resulting in a net contraction of 2 inches. Since we printed the silicone heart to scale, we then graphed desired position of the end of the rack for 1 period. We first plotted different beat per minute values (aka different period values), and then drew a generalized graph as shown below.

Because we don't have a position sensor in our system, we reverse engineer the force by using the RPM of the motor, the circumference of the gear, and the distance that the rack needs to travel, and the time it has to travel that distance (the last two points were determined as discussed above). This equation was directly translated into our code.

As mentioned above, we wanted our rack and pinion to be in contact with the heart for most of its travel, which meant that we could then treat the system similarly to a render of a soft wall, where the silicone heart represented a wall with a certain stiffness. We went through many iterations of firmware, working through hardware bring up, signal processing, and different ways to compute and render the heartbeat pattern. Our final system did not have a position sensor; we instead had physical stops on our rack and pinion, and would adjust the BPM based on the heartbeat recorded from

Our final code operates as follows: Upon start up, the optical sensor initializes and turns on its IR LED. On every loop it takes a measurement of reflectivity, which when measuring a fingertip, is tied to the heart rate. We have a hysteresis threshold which waits for the values to climb above a certain value to indicate that we’re approaching the peak of a cycle. Once we go back over the hysteresis threshold, we take the time at the max point and subtract the time from the last max point to get a time for period, and also calculate the beats per minute. We then, having previously empirically found a compelling multiplier for force, go through a pattern of motor actuation patterns, using the hapkit force/PWM functions and a timer to render the feeling of the “lub-dub” of a heart beat. We use the period of the heart cycle to adjust the rate of beating, and PWM to adjust the force between the lub and the dub.

A day before the project, we noticed that the hapkit’s PWM control was interfering with the heartbeat sensor. While sensor measurements hovered in a range of values comparable to a normal heartbeat, they were not actually changing based on biofeedback. We tried to adjust both the PWM control code as well as the Max30105 library/settings but weren’t able to avoid sensor data not working when they were run together. These systems both were tested and worked when run separately, and so we decided to adjust our code to run on two hapkits: one would manage the optical sensor, and included a one-way communication pin that would transmit the data from the sensor to the second hapkit, which calculated the actuation pattern and controlled the motors. We ran into issues with transmitting and receiving data, and weren’t able to successfully send the data between the two boards by demo time.

Demonstration / application

On Demo Day, we had two computers set up to handle the heart sensor and the motor actuation for the heartbeat. As a user sat down, they placed their right index finger on the sensor and began sensing the BPM. Then we placed the headphones on the user. The user grabbed and held the heart in their left hand. In the first part of the demonstration, we played an aggressive metal song, “The House of Wolves” by Bring Me the Horizon. We watch to see how the song affects the user’s heart rate for about a minute and the heart changed to simulate the same BPM. Then for the second portion of the demonstration, we play a slow meditation song, “Bouncing Clouds” by Sava Sol. Again, we watch to see how the song affects the user’s heart rate for about a minute and again the heart changed to simulate the same BPM.

Results

Many users at the open house were able to sit down and go through the demonstration. Upon first picking up the heart, we observed that many users heart rates spiked. This could be from the shock factor of feeling a beating heart in their hands. On average, most users’ heart rates went up when they were listening to the screamo song, and the users’ heart rates went down when they were listening to the calming song. This aligns with the research about how music deeply affects a person’s heart rate. In the future, we would like to quantify this more thoroughly.

We also found that when the heart stopped to re-calibrate, many users’ heart rates also spiked from the shock of their heart “not working anymore.”

Future Work

An interesting take on this project would be to program the silicone heart to beat at a false heart rate and see if the user’s actual heart rate believes the haptic feedback heart in their hand. It has been shown that visual cues play a large role in how users perceive the world even though it’s not always true. For example, in our hapkits the proxy not going beyond the wall when in reality the motor isn’t strong enough to keep the handle at the wall.

While we tried to make a compelling heart, further improvements could revolve around testing the effects of weight, material durometer, and actuation styles in creating a compelling heart. A study could be created in comparing how compelling a series of a heartbeat actuation pattern can be.

Acknowledgments

Thank you to Margaret Koehler and Nathan Usevitch for the silicone mold for the heart.

Files

Our component list can be found at: https://docs.google.com/spreadsheets/d/1s_iWHLjz8uhi6J0CMJ4Up_CAkWWR13BXFL09EyfxIjg/edit?usp=sharing

Final Motor Code: Attach:HB_final_M.zip

Final Sensor Code: Attach:HB_final_s.zip

References

Fu, Michael C., et al. "Haptic biofeedback for improving compliance with lower-extremity partial weight bearing." Orthopedics 37.11 (2014): e993-e998.

Gammill, Stephen L., et al. "Cardiac measurements in systole and diastole." Radiology 94.1 (1970): 115-120.

Giggins O., Persson U., Caulfield B. Biofeedback in Rehabilitation. Journal of NeuroEngineering and Rehabilitation 2013, 10:60 jneuroengrehab.biomedcentral.com/track/pdf/10.1186/1743-0003-10-60

Janidarmian, Majid, et al. "Haptic feedback and human performance in a wearable sensor system." 2016 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI). IEEE, 2016. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7455975

Koelsch, Stefan, and Lutz Jäncke. "Music and the heart." European heart journal 36.44 (2015): 3043-3049.

---

Appendix: Project Checkpoints

Checkpoint 1

Checkpoint 1 Deliverables:

1. Connect a heart beat monitor to an Arduino and output a graph of the user’s heartbeat. Within this are the following sub-tasks:
   1. Choose and acquire Heart rate sensor
   2. Test sensor locations; Create something to hold sensor in place (get stable reading)
2. Program a set of buttons so that when a user presses a button, a pre-recorded rhythm (abnormal rhythm, normal rhythm, fast vs slow heartbeat) is displayed on the output graph. Within this are the following sub-tasks:
   1. Interface buttons to Arduino (Arduino does A when button A is pressed)
   2. Collect Heart rate data (especially unusual ones like exercise)

For our heartbeat sensor, we decided to use the Sparkfun MAX30105 breakout board, which houses an optical sensor that can use red, green, and IR LEDs to measure the changing in reflectivity of a surface. If something is kept close and still enough, it can optically measure a heartbeat. This breakout board also has an arduino library to handle initialization, as well as documentation on how to use it to measure a heartbeat.

Using our hapkit microcontroller, we connected the breakout board and used the libraries to set up a sketch that would plot reflectivity results. We played around with different ways to record a measurement, but eventually settled on using pipe cleaners to hold the sensor either against the index finger, or to hold the sensor against the inside of the wrist. In both cases we were able to get a PPG signal that had the correct waveform and a measurable, stable amplitude and frequency.

Our test setup for checkpoint 1 (left) and close-up of sensor attached to a finger with pipe cleaner (right)

We needed to do some cleaning of the output data and some adjustments, as demonstrated in the output graphs below.

signal data with slight motion in the finger shows drift in vertical offset of the signal (top) and signal data with the optical sensor attached and the finger and arm both resting on a table (bottom)

In order to trigger the button presses for the pre-recorded signals, we decided to set up 4 buttons across the hapduino's D0-D4 pins. The buttons are normally open, and we used a pull down resistor to tie them to ground. When pressed, the digital input reads HIGH. In the sketch's main loop, we watch for when the button state to change from LOW to HIGH and then would play the pre-recorded heartbeat pattern.

We ran into some issues finding pre-recorded heartbeats to trigger the buttons with. There were two main approaches we brainstormed and tested:

• The first was to find a CSV of example heartbeat data and display those points to the serial plotter screen on button press. While this removed the burden of creating the information from us, we realized that we would run into issues adjusting the signals to display them on the serial plotter, particularly from adjusting the frequency. We found a few datasets showing heartbeat information, but it was mainly regular heartbeat data and over long periods of time (e.g. Holder Monitors over 24 hours). See the datasets here:
• The other idea was to characterize a single cycle of a wave in an array, and then when a button is pressed, adjust the output of that array based on our desired frequency, offset, and amplitude. We would use the same array of a single cycle to play 4 all four pre-recorded heartbeats. In this case, we would have to create this array of data, but by characterizing its behavior, we'd be able to more easily display it. This would unfortunately add a lot of complexity to our sketch.
• Lastly, we found a few images of EKG plots of unusual heartbeats, from which we could extrapolate datapoints and create our own CSV file. See the link here: https://www.sfcollege.edu/Assets/sf/ips/pdfs/EKG%20Flash%20Cards.pdf.

We didn't manage to display the pre-recorded heartbeats, but are currently diving into the issue of how to best display them in our project. We have a couple samples of heart data for our cases, and will hopefully integrate this very soon. Along the way, we also discovered that the Arduino serial plotter is very limited in its functionality, and that we'd want to use processing for graphics. Because the MAX30105 example code used serial plotter (and its syntax is the same as using Processing), we stuck with serial plotter for now. Before our next checkpoint we'll write Processing code to handle our signal output. This week, we did get a headstart on the silicone envelope which is a part of the final deliverable. We printed a new (scaled-down) internal piece of the mold, thus making the heart walls much thicker and distributing the force of the motor across a person’s hand.

Checkpoint 2

Checkpoint 1 Deliverables:

1. Connecting the 1 or 2 motors to the arduino, and having the motors move left/right in motion with the input pattern, for both the recorded buttons and the user’s heartbeat.

Hardware Development

As we discussed in checkpoint 1, we 3D printed a smaller core for the silicone heart mold. This week, we silicone cast the heart to have a ~0.75" thickness and now the heart has a heftiness to it when you hold it in your hand.

We developed a motor mount that houses the motor. The mount has a flange that sits at the mouth of the cavity inside of the cast silicone heart. Also, we lasercut and 3D printed rack and pinion mechanisms that is actuated by the motor. The motor mount has a mount and stopper system incorporated into it for the racks to slide back and forth. The first iteration was short and we realized that we wanted the motor to be situated deeper inside of the heart cavity. We prototyped a second iteration of the motor mount and that is deeper inside.

We started integrating the rack and pinion with the arduino code. The testing has lead to some realistic heart beats and we have concluded that we need only one motor inside of the heart.

For the sensor, we prototyped various ways in which the finger could rest on the sensor. We tried rubber bands, pipe cleaners, and velcro. We determined that velcro was the most comfortable and secure for various finger types.

Software Development

We integrated software that takes in the heart rate reading and standardized it accordingly to a moving average of the most recent values so that it doesn't have a drifting average over time. The DC component is removed, leaving a heart rate centered at zero. Since it is a wave centered around zero, the code now calculates beats per minute (BPM). It does this through a hysteresis threshold at the top and bottom to account for noise, drift, and amplitude. The code does a weighted average to report a weighted average of the BPM over time.

Even though we have this signal, we cannot send this directly to the motor. We decided to take an array of clean values of the heartbeat and modify it according to a bpm max, and min, that are the inputs from the user. We calculate the max and min over a time period of 2 seconds. This allows for us to create a function that generates a force that the motor needs to produce at any given time. It takes the current time and finds the next point and previous point in the discrete "real" data we have and extrapolates a line between them to approximate the force needed at every time step.

Project Song