Skip to main navigation Skip to search Skip to main content

Tactile Emotions: Multimodal Affective Captioning with Haptics Improves Narrative Engagement for d/Deaf and Hard-of-Hearing Viewers

  • Caluã De Lacerda Pataca
  • , Saad Hassan
  • , Lloyd May
  • , Michelle M. Olson
  • , Toni D'Aurio
  • , Roshan L. Peiris
  • , Matt Huenerfauth

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

Abstract

This paper explores a multimodal approach for translating emotional cues present in speech, designed with Deaf and Hard-of-Hearing (dhh) individuals in mind. Prior work has focused on visual cues applied to captions, successfully conveying whether a speaker's words have a negative or positive tone (valence), but with mixed results regarding the intensity (arousal) of these emotions. We propose a novel method using haptic feedback to communicate a speaker's arousal levels through vibrations on a wrist-worn device. In a formative study with 16 dhh participants, we tested six haptic patterns and found that participants preferred single per-word vibrations at 75 Hz to encode arousal. In a follow-up study with 27 dhh participants, this pattern was paired with visual cues, and narrative engagement with audio-visual content was measured. Results indicate that combining haptics with visuals significantly increased engagement compared to a conventional captioning baseline and a visuals-only affective captioning style.

Original languageEnglish
Title of host publicationProceedings of the 2025 CHI Conference on Human Factors in Computing Systems
EditorsBongshin Lee, Marshini Chetty, Phoebe Toups-Dugas
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages17
ISBN (Electronic)9798400713941
DOIs
Publication statusPublished - 2025
Externally publishedYes
EventInternational Conference on Human Factors in Computing Systems 2025 - Yokohama, Japan
Duration: 26 Apr 2025 → 1 May 2025
https://chi2025.acm.org (Website)
https://dl.acm.org/doi/proceedings/10.1145/3706599 (Proceedings - Extended Abstracts)
https://dl.acm.org/doi/proceedings/10.1145/3706598 (Proceedings)

Conference

ConferenceInternational Conference on Human Factors in Computing Systems 2025
Abbreviated titleCHI 2025
Country/TerritoryJapan
CityYokohama
Period26/04/25 → 1/05/25
Internet address

Keywords

  • Accessibility
  • Emotion / Affective Computing
  • Empirical study that tells us about how people use a system
  • Individuals with Disabilities & Assistive Technologies

Cite this