Noel
Caption: The final vibrotactile
glove design.
Spatial Exploration with Vibration Feedback
Project team member(s): Noel Spurgeon
The topic of this project will be to create a low-cost, easily configurable haptic educational device that utilizes vibrotactile feedback at the extremes of the body in order to convey information about physics or mathematical simulations. The device consists of a glove, upon which an inertial measurement unit and two vibrotactile sensors will be mounted. The IMU will track the position of the hand, and the vibrotactile sensors will provide feedback based on hand position and information about the two-dimensional space to be explored. The motivation for this project is that visual methods of learning about topics like magnetic fields are often confusing to a student, and perhaps tactile stimulation will help students grasp a more intuitive understanding of these topics.
On this page... (hide)
Introduction
In recent years, the integration of haptic devices into educational environments has become an increasingly popular area of research. Although the use of haptic devices such as dental and surgical simulators is well-established in advanced educational settings, the focus on introducing haptic feedback to the high school and younger age bracket is a relatively new development. While many of the devices being studied are force-feedback based such as the Phantom Omni and Novint Falcon, there has also been investigation into the efficacy of vibrotactile haptic feedback in education. This area of research has shown promise as an alternative to force-feedback devices.
The goal of this project is to create a generalized, wearable haptic educational device using two vibrotactile actuators on the mounted on the top and the bottom index finger to provide feedback based on the position of the user's hand in 2-dimensional space. The device essentially has a free workspace, allowing for the scaling of the haptic environment as required by the educator. Utilizing the extremities as an input area also expands the range of the device.
This device aims to be utilized in conjunction with visual feedback or physical props to teach concepts about information distributed over a large area, such as temperature gradients or magnetic fields, in order to reinforce the information. In this case, a vibrotactile approach is appropriate because it allows the user unconstrained exploration of an environment, and because the environment is not size-limited, details can be exaggerated and magnified for emphasis without crowding the workspace. In addition, direction of a force can also be conveyed by utilizing either the top or bottom vibrotactor, something which is difficult to convey in a force-feedback device without feeling somewhat unnatural for surface exploration.
Background
Haptic feedback can provide both an increase in performance and student interest for various educational topics. In a 2006 study by Jones et. al., the effect of haptic feedback on the educational experience of middle and high school students was studied through the use of a program on viruses and nanoscale science. Three different input devices were utilized (a mouse, a haptic joystick, and a Phantom Omni), and it was discovered that haptic feedback caused students to be more interested and engaged in the program, and resulted in an increase in the number of haptic terms used to describe the information learned, indicating an integration of the visual information with the haptic feedback (Jones et. al.). In addition, a study involving the HaptEK16 visuo-haptic physics simulation have indicated significantly higher test scores and levels of engagement in students who received the complementary instruction with the haptic interface than those who did not (Hamza-Lup and Adams).
In the study of vibrotactile feedback as an educational mechanism, an experiment by Kim et. al. has demonstrated that there is no significant difference in student learning between kinesthetic and haptic feedback. A group of eighth-grade Korean students were educated about the Coriolis Effect using one of two haptically-enhanced presentations (with either a Phantom Omni or vibrotactile glove), or a presentation containing only audiovisual information. While there was no significant difference in post-presentation assessment scores between the two haptic presentations, the test scores of those students who had received haptic information were significantly higher than those who had not.
In addition, construction of a vibrotactile glove mouse by Park et. al. has shown promise as a device. The glove, which utilizes a wrist-mounted accelerometer and a Bluetooth module, reads the movement of the user's wrist and corrects the accelerometer readings on the computer, and then sends instructions to the six vibrotactile devices within the glove. The glove can be used to move the cursor on the screen, as well as to ‘feel' patterns and shapes (Park et. al.).
Methods
Hardware design and implementation
The chief electronic hardware components of this device were the MPU-6050 IMU and breakout board (part number SEN-11028 from SparkFun electronics), the Arduino Uno microcontroller, the Ardumoto shield, and 2 Precision Microdrives 310-101 vibration motors. Data sheets and/or schematics for each of these components can be downloaded in the ‘files' section. Additional hardware components included assorted wires, solder, fabric, and the Isotoner glove the device was mounted on. The glove was chosen specifically for its tight fit, as well as the thickness of the fabric, which played a part in damping some of the vibrations from the motors so that they would not introduce noise into the IMU measurements.
The Arduino Uno, the Ardumoto Shield, and the IMU breakout board are shown below.
The Ardumoto shield was used to provide sufficient current at 5V to the two vibration motors, off of PWM pins 3 and 11 of the Arduino Uno. Although the vibration motors were rated to run at up to 3.6 V RMS, voltage dividers were utilized to keep the motors to the lower end of their rated range (2V RMS) after it was found that the vibrations output at the high end of the operating range were too large, propagating down the user's hand and causing additional noise in the IMU. The voltage dividers utilized 150 Ω and 560 Ω resistors, arrayed in the configuration shown below:
The IMU breakout board was then connected to the pins of the Arduino Uno. The physical connections from the breakout board to the Uno are as follows:
- VDD - Arduino 3.3v
- GND - Arduino GND
- INT - Arduino digital pin 2
- FSYNC – No connection
- SCL - Arduino Analog Pin 5
- SDA – Arduino Analog Pin 4
- VIO - Arduino 3.3v
- CLK – No connection
- ASCL – No connection
- ASDA – No connection
The IMU breakout board was then sewn to a square of woven nylon fabric, using the two holes opposite the connections and standard nylon thread. The fabric square was then attached to the Isotoner glove using low-temperature hot glue. The decision to mount the breakout board on additional fabric was due to the amount of the stretch in the glove: repeated stretching of the threads connecting the breakout board to the glove would weaken the threads and decrease the robustness of the device.
Finally, the vibration motors were secured to the top and bottom of the inside of the index finger of the glove using low-temperature hot glue. The motor wires were left inside the glove to decrease the risk of being caught on surrounding objects. A picture of the completed project can be seen below:
Software
As the IMU contains only accelerometers and gyroscopes, there was no way to directly track the position of the user's hand. Therefore, position had to be determined via integration of the various components of acceleration. A trapezoidal integration scheme was used, with a time step of 0.01 seconds. This was due to a hardware limitation instead of a programming decision.
3rd-party digital motion processing (DMP) code was used to determine the real-world X, Y, and Z values of acceleration, compensated for gravity and orientation of the IMU. A link to the GitHub page for this code can be found in the 'references' section. Because the accelerometer had a ‘settling' time, additional code was implemented to display appropriate feedback when the values had stabilized.
While an unfiltered integration approach during flat-surface testing worked well in terms of calculating position with a reasonable amount of noise, human hand tremors in the wearable device increased the noise levels to a degree that the position tracking was no longer accurate to any reasonable degree. Therefore, the decision was made to filter the acceleration, the velocity, and the position measurements using a simple filter:
X_filtered = filterparameter*X_previous + (1-filterparameter)*X
where the filter parameter was 0.99, as fairly ‘stiff' filtering was required to damp out the noise. This scheme worked well for large, slow movements.
After the X and Y position were determined, these positions were fed into a function that determined a PWM value for the motor, as well as determining which motor should be turned on. For the demonstration, this function was based on the difference between the X and Y positions. The larger the difference, the larger the amplitude of the vibration, and the larger the PWM value. If the magnitude of the X position was greater than that of the Y position, the ‘force' was considered to be negative, and the bottom vibrotactor was powered. If the Y position as larger, the ‘force' was considered to be positive, and the top vibrotactor was powered.
Code can be found in the ‘files' section.
Results
Overall, the device worked fairly well for slow, large movements, and generated sensations that were suitably compelling. The filtering, although it reduced a good deal of the noise and made the position measurements much more stable, effectively negated the device's ability to respond to rapid accelerations and changes in direction. Given that the device is exploratory in nature, the movements of the users in general were fairly slow, rendering this issue smaller than it would be otherwise.
In addition to the substantial noise, the issue of the accelerometer settling time negatively affected the user experience. While the acceleration values were settling, the user needed to remain extremely still, with their hand on a flat surface. Because this process took a minute to a minute and a half, most users grew impatient with the wait, and began to move around before the settling process was complete. Since the portion of the code that dealt with the settling time relied on accelerometer values to determine if settling had taken place, any movement on the part of the user could cause interference, which occasionally led to the device getting ‘stuck' in the waiting phase.
Due to the DMP algorithms used, any substantial yaw/pitch/roll motion of the user's hand would cause a substantial change in the acceleration, creating the problem of ‘runaway position', for which the accelerometer had no way of compensating. However, once users were instructed to keep their hand flat, this problem was largely resolved, as the first impulse of many users was to flip their hand upside down, which was an unforeseen reaction to the device.
Finally, some users expressed discomfort at the tightness of the glove, and some difficulty in arranging the vibrotactors to be on the top and bottom of the index finger, due to the fact the motors were attached to the lining of the glove, and the lining was free to rotate. This was especially a problem in participants with larger hands. However, the tightness of the glove was instrumental in maintaining the position of the vibrotactors and IMU once it had been properly fitted, and there was little to no slippage during the trials.
Future Work
The efficacy of this kind of feedback could be tested in a fashion similar to that of the work done by Kim et. al. Students would receive haptic and non-haptic instruction on a topic such as magnetic fields, and the haptic instruction would consist either of a 3-DOF haptic device, or the vibrotactile glove. Attitudes toward the material and knowledge about the material could be compared before and after the instruction, to determine if the device were a valuable educational tool.
There are several areas in which this device could be improved for the future. First, research into improved methods of motion tracking would increase the device's sensitivity and remove the issues with rapid changes in direction and speed, allowing for quicker exploratory movements. As the drawbacks in using integrated accelerometer data have become quite apparent in this project, the inclusion of another method of motion tracking, such as an optical sensor, may be a valid solution.
Also, correction of the yaw/pitch/roll issues would greatly improve user experience. The yaw-pitch-roll issues could likely be solved by incorporating measurements of these values into the algorithm which outputs acceleration, or with the addition of a magnometer or optical tracker. The user experience surrounding the accelerometer stabilization time, could also be improved. The accelerometers could be replaced with ones that have a shorter time stabilization time constant, or a more noticeable visual indicator, such as a light that switches on when the system is ready, could be incorporated to encourage the user to be more patient.
Also, some of the construction techniques could be altered in order to make the system more robust. The soldered connections between the wires for the IMU breakout board and the Arduino had unforeseen issues with sturdiness during the open house which were not revealed during initial testing. Although the connections remained intact, there was substantial bending of the connective wires over the course of the demonstration due to the differently-sized hands of the users, which could lead to eventual failure of the device. Therefore, a more robust system of connections should be designed in order to increase device longevity and durability.
In addition to additional user and educator testing to resolve comfort and sturdiness issues, a GUI should be developed for the programming of haptic environments. This would allow educators to create simulations with little previous programming knowledge, which would help to make the device more accessible to a greater variety of people. This would allow it to fulfill its intended use as a supplement to audio-visual educational material in middle and high school classrooms.
Acknowledgments
Amy Ahearn (LDT program)
Files
Attach:motor_datasheet_noel.pdf
References
Jones, M. G., Minogue, J., Tretter, T. R., Negishi, A. and Taylor, R. Haptic augmentation of science instruction: Does touch matter?. Sci. Ed., 90: 111–123, 2006.
Hamza-Lup, F.G., Adams, M.. Feel the pressure: e-Learning system with haptic feedback. The 16th symposium on haptic interfaces for virtual environments and teleoperator systems (Haptics 2008), 445–450, March 13–14, Reno, Nevada (2008)
Park, J., Lee, H., Jeong, J., and Jang, T., "Realization of a vibro-tactile glove type mouse," … of the 16th ACM Symposium on …, vol. 1, no. 212, pp. 255–256, 2009.
Kim, Y. Park, S. Kim, H. Jeong, H. Ryu, J. , "Effects of different haptic modalities on students' understanding of physical phenomena," World Haptics Conference (WHC), 2011 IEEE , vol., no., pp.379-384, 21-24 June 2011.
Appendix: Project Checkpoints
Checkpoint 1
Thus far, everything that has been set out to be accomplished by the first checkpoint in the project has been accomplished. The master's students from the teaching program have been contacted, and interviews are currently being scheduled to determine what kind of lesson plans this device will be useful for, and to consider these lesson plans in the design of the device. In addition, all of the electronic components for the device have been ordered, consisting of two vibrotactile units and an IMU from Arduino. While the vibrotactile units were not difficult to procure (and were very inexpensive), the IMU required more detailed research in order to avoid spending excessive amounts of money on a component that may not be adequate.
Checkpoint 2
Unfortunately, all goals for this checkpoint were not achieved. Due to delays in component shipment, work on the physical portion of the project was not started until late in the week. Although the IMU and one of the vibrotactile components have both been tested and are somewhat functional independently, they still have yet to be combined into a functional system. In addition, the IMU was slightly more challenging to work with than originally thought, and the limited number of vibration outputs for the Immersion daughterboard have made generating a compelling sensations somewhat more difficult than originally planned.
Checkpoint 3
- Complete final construction
- Further refinement and testing
Although the hardware and software are largely functional for the device, it still falls short of completion goals. The LRMs and IMU are working well in concert, and the motion tracking from the IMU has been accomplished to a point where it can be used to generate meaningful feedback, although problems with drift were encountered initially. These were likely due to vibrations from the LRM, as the IMU was placed atop the actuator for testing haptic feedback. The electrical design is completely planned out on the breadboard.
An in-progress photo of the final electrical configuration. In the current configuration, the second LRM is wired to the PWMB pins of the Ardumoto board.
Several changes have taken place over the past week with regards to the original design. The idea of using the Immersion daughterboard was abandoned in favor of using the PWM pins on the Ardumoto shield, as these could provide adequate current to drive the actuators and did not rely on a predetermined library of haptic effects to give feedback to the user, allowing for greater customization. In addition, the location of the LRMs has changed: through some initial testing, it was found that the vibrations were more compelling if they were moved to the tip of the index finger instead of being placed in the center of the palm and back of the hand. This difference in sensation was especially noticeable for the vibrotactile device on the backside of the hand, as placing the LRM on the fingernail created a suitably different sensation from the LRM placed on the other side, increasing the efficacy of the device. This, however, will require a redesign of the device to be less like a band and more like a glove. The IMU will be mounted on the back of the glove, as the workspace is large enough that if the hand is kept in a ‘pointing' configuration, the differences between the displacement of the finger and the displacement of the hand will not be large enough to cause problems.
The algorithm used to determine what vibrotactile feedback is produced utilizes a conversion to determine the value for the PWM used to drive each haptic device. First, the magnitude and direction of the information which needs to be conveyed (force, in the case of the demo) is calculated based on the displacement of the IMU. This force is then converted into a 0-255 integer using a conversion factor dependent on both the workspace and the range of forces to be experienced. This integer is then used as the input for the PWM of the correct LRM, depending on the direction of said force. Difficulties encountered with this portion of the project include the problems with drift, as well as a burnt-out LRM due to incorrect hookup of a voltage divider.
The tasks that remain before Tuesday are to transfer the electrical components to a more robust (i.e. permanent) configuration, and to create the glove to hold the LRMs and IMU on the user's hand.