Written and multimodal representations as predictors of expertise and problem-solving success in mathematics

Sharon Oviatt, Adrienne Cohen

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

14 Citations (Scopus)


One aim of multimodal learning analytics is to analyze rich natural communication modalities to identify domain expertise and learning rapidly and reliably. In this research, written and multimodal representations are analyzed from the Math Data Corpus, which involves multimodal data (digital pen, speech, images) on collaborating students as they solve math problems. Findings reveal that in 96-97% of cases the correctness of a group's solution was predictable in advance based on students' written work content. In addition, a linear regression revealed that 65% of the variance in individual students' domain expertise rankings could be accounted for based on their written work content. A multimodal content analysis based on both written and spoken input correctly predicted the dominant domain expert in a group 100% of the time, exceeding unimodal prediction rates. Further analysis revealed a reversal between experts and non-experts in the percentage of time that a match versus mismatch was present between their oral and written answer contributions, with non-experts demonstrating higher mismatches. Implications are discussed for developing reliable multimodal learning analytics systems that incorporate digital pen input to automatically track consolidation of domain expertise.

Original languageEnglish
Title of host publicationICMI'13 - Proceedings of the 15th ACM on International conference on multimodal interaction
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages7
ISBN (Print)9781450321297
Publication statusPublished - 2013
Externally publishedYes
EventInternational Conference on Multimodal Interfaces 2013 - Coogee Bay Hotel, Sydney, Australia
Duration: 9 Dec 201313 Dec 2013
Conference number: 15th
https://icmi.acm.org/2013/ (Conference website)
https://dl.acm.org/citation.cfm?id=2522848&picked=prox (ACM proceedings)


ConferenceInternational Conference on Multimodal Interfaces 2013
Abbreviated titleICMI 2013
Internet address


  • collaborative problem solving
  • digital pen & writing
  • domain expertise
  • math data corpus
  • multimodal learning analytics
  • prediction

Cite this