Youjin Choi
Gwangju Institute of Science and Technology, Gwangju, Korea, Republic of
Deaf and hard of hearing (DHH) people enjoy music and access it using a music-sensory substitution system that delivers sound together with the corresponding visual and tactile feedback. However, it is often challenging for them to comprehend the colorful visuals and strong vibrations that are designed to represent music. We confirmed that it is necessary to conceptualize cross-modal mapping before experiencing music sensory substitution through focus group interviews with 24 DHH people. To improve the music appreciation experience, a cross-modal music conceptualization system was implemented herein, which is a prototype that allows DHH people to explore the visuals and vibrations associated with music to perceive and appreciate. An evaluation with 28 DHH individuals demonstrated the capability of the system to improve subjective music appreciation experience via music-sensory substitution. Eventually, DHH people with negative attitudes toward music became positive in the exploration and customization process with our system.
doi.org/10.1145/3613904.3642665
The ACM CHI Conference on Human Factors in Computing Systems (https://chi2024.acm.org/)