Exploring students' cognitive and affective states during problem solving through multimodal data: lessons learned from a programming activity

Background: Problem-solving is a multidimensional and dynamic process that requires and interlinks cognitive, metacognitive, and affective dimensions of learning. However, current approaches practiced in computing education research (CER) are not sufficient to capture information beyond the basic programming process data (i.e., IDE-log data). Therefore, how cognition and affect intertwine and unfold over time in programming problem-solving activities are rarely investigated. Objectives: In this study, we examined how the theory-informed measures from multimodal data that we have selected as proxies for cognitive and affective dimensions of learning, are associated with student performance, and in comparison, to prior-knowledge. Methods: A high-frequency temporal data was collected with a camera, an electroencephalogram, and an eye-tracker from 40 computer science students (bachelor and master studies) in the context of a code-debugging activity. To study the cognitive processes associated with learning we focused on cognitive load theory (CLT) and the human information processing model. In addition, we complemented CLT with the model of affective dynamics in learning to avoid the machine reductionism perspective. Results: Our findings demonstrated that attention, convergent thinking, and frustration were positively correlated with students' successful code-debugging (i.e., performance), and frequently manifested by high performing participants. Cognitive load, memory load, and boredom were negatively correlated with students' performance, and typically manifested by low performing participants. Implications: Extending the context of analysis in reference to student cognitive processes and affective states, affords educators not just to identify lower performers, but also to understand the potential reasons behind their performance, making our method an important contribution in the confluence of CER and the learning technology communities. In addition, the insights extracted from our analyses allow us to discuss potential avenues for improving learning design and the design of interactive learning systems to support the development of students' problem-solving skills.