Abstract
This paper explores a multimodal approach for translating emotional cues present in speech, designed with Deaf and Hard-of-Hearing (dhh) individuals in mind. Prior work has focused on visual cues applied to captions, successfully conveying whether a speaker's words have a negative or positive tone (valence), but with mixed results regarding the intensity (arousal) of these emotions. We propose a novel method using haptic feedback to communicate a speaker's arousal levels through vibrations on a wrist-worn device. In a formative study with 16 dhh participants, we tested six haptic patterns and found that participants preferred single per-word vibrations at 75 Hz to encode arousal. In a follow-up study with 27 dhh participants, this pattern was paired with visual cues, and narrative engagement with audio-visual content was measured. Results indicate that combining haptics with visuals significantly increased engagement compared to a conventional captioning baseline and a visuals-only affective captioning style.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems |
| Editors | Bongshin Lee, Marshini Chetty, Phoebe Toups-Dugas |
| Place of Publication | New York NY USA |
| Publisher | Association for Computing Machinery (ACM) |
| Number of pages | 17 |
| ISBN (Electronic) | 9798400713941 |
| DOIs | |
| Publication status | Published - 2025 |
| Externally published | Yes |
| Event | International Conference on Human Factors in Computing Systems 2025 - Yokohama, Japan Duration: 26 Apr 2025 → 1 May 2025 https://chi2025.acm.org (Website) https://dl.acm.org/doi/proceedings/10.1145/3706599 (Proceedings - Extended Abstracts) https://dl.acm.org/doi/proceedings/10.1145/3706598 (Proceedings) |
Conference
| Conference | International Conference on Human Factors in Computing Systems 2025 |
|---|---|
| Abbreviated title | CHI 2025 |
| Country/Territory | Japan |
| City | Yokohama |
| Period | 26/04/25 → 1/05/25 |
| Internet address |
|
Keywords
- Accessibility
- Emotion / Affective Computing
- Empirical study that tells us about how people use a system
- Individuals with Disabilities & Assistive Technologies
Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver