Counting clicks is not enough: validating a theorized model of engagement in learning analytics

Ed Fincham, Srećko Joksimović, Alexander Whitelock-Wainwright, Jan Paul Van Staalduinen, Vitomir Kovanović, Dragan Gašević

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

Student engagement is often considered an overarching construct in educational research and practice. Though frequently employed in the learning analytics literature, engagement has been subjected to a variety of interpretations and there is little consensus regarding the very definition of the construct. This raises grave concerns with regards to construct validity: namely, do these varied metrics measure the same thing? To address such concerns, this paper proposes, quantifies, and validates a model of engagement which is both grounded in the theoretical literature and described by common metrics drawn from the field of learning analytics. To identify a latent variable structure in our data we used exploratory factor analysis and validated the derived model on a separate sub-sample of our data using confirmatory factor analysis. To analyze the associations between our latent variables and student outcomes, a structural equation model was fitted, and the validity of this model across different course settings was assessed using MIMIC modeling. Across different domains, the broad consistency of our model with the theoretical literature suggest a mechanism that may be used to inform both interventions and course design.

Original languageEnglish
Title of host publicationProceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK'19)
Subtitle of host publicationLearning Analytics to Promote Inclusion and Success
EditorsChristopher Brooks, Rebecca Ferguson, Ulrich Hoppe
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages501-510
Number of pages10
ISBN (Electronic)9781450362566
DOIs
Publication statusPublished - 2019
EventInternational Learning Analytics & Knowledge Conference 2019 - Arizona State University, Tempe, United States
Duration: 4 Mar 20198 Mar 2019
Conference number: 9th
https://lak19.solaresearch.org/

Publication series

NameACM International Conference Proceeding Series

Conference

ConferenceInternational Learning Analytics & Knowledge Conference 2019
Abbreviated titleLAK 2019
CountryUnited States
CityTempe
Period4/03/198/03/19
Internet address

Keywords

  • Engagement
  • Factor analysis
  • Measurement invariance
  • MOOCs
  • Structural equation modeling

Cite this

Fincham, E., Joksimović, S., Whitelock-Wainwright, A., Van Staalduinen, J. P., Kovanović, V., & Gašević, D. (2019). Counting clicks is not enough: validating a theorized model of engagement in learning analytics. In C. Brooks, R. Ferguson, & U. Hoppe (Eds.), Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK'19): Learning Analytics to Promote Inclusion and Success (pp. 501-510). (ACM International Conference Proceeding Series). New York NY USA: Association for Computing Machinery (ACM). https://doi.org/10.1145/3303772.3303775
Fincham, Ed ; Joksimović, Srećko ; Whitelock-Wainwright, Alexander ; Van Staalduinen, Jan Paul ; Kovanović, Vitomir ; Gašević, Dragan. / Counting clicks is not enough : validating a theorized model of engagement in learning analytics. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK'19): Learning Analytics to Promote Inclusion and Success. editor / Christopher Brooks ; Rebecca Ferguson ; Ulrich Hoppe. New York NY USA : Association for Computing Machinery (ACM), 2019. pp. 501-510 (ACM International Conference Proceeding Series).
@inproceedings{5935d0e21820437290a878ff2bcfe3ac,
title = "Counting clicks is not enough: validating a theorized model of engagement in learning analytics",
abstract = "Student engagement is often considered an overarching construct in educational research and practice. Though frequently employed in the learning analytics literature, engagement has been subjected to a variety of interpretations and there is little consensus regarding the very definition of the construct. This raises grave concerns with regards to construct validity: namely, do these varied metrics measure the same thing? To address such concerns, this paper proposes, quantifies, and validates a model of engagement which is both grounded in the theoretical literature and described by common metrics drawn from the field of learning analytics. To identify a latent variable structure in our data we used exploratory factor analysis and validated the derived model on a separate sub-sample of our data using confirmatory factor analysis. To analyze the associations between our latent variables and student outcomes, a structural equation model was fitted, and the validity of this model across different course settings was assessed using MIMIC modeling. Across different domains, the broad consistency of our model with the theoretical literature suggest a mechanism that may be used to inform both interventions and course design.",
keywords = "Engagement, Factor analysis, Measurement invariance, MOOCs, Structural equation modeling",
author = "Ed Fincham and Srećko Joksimović and Alexander Whitelock-Wainwright and {Van Staalduinen}, {Jan Paul} and Vitomir Kovanović and Dragan Gašević",
year = "2019",
doi = "10.1145/3303772.3303775",
language = "English",
series = "ACM International Conference Proceeding Series",
publisher = "Association for Computing Machinery (ACM)",
pages = "501--510",
editor = "Christopher Brooks and Rebecca Ferguson and Ulrich Hoppe",
booktitle = "Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK'19)",
address = "United States",

}

Fincham, E, Joksimović, S, Whitelock-Wainwright, A, Van Staalduinen, JP, Kovanović, V & Gašević, D 2019, Counting clicks is not enough: validating a theorized model of engagement in learning analytics. in C Brooks, R Ferguson & U Hoppe (eds), Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK'19): Learning Analytics to Promote Inclusion and Success. ACM International Conference Proceeding Series, Association for Computing Machinery (ACM), New York NY USA, pp. 501-510, International Learning Analytics & Knowledge Conference 2019, Tempe, United States, 4/03/19. https://doi.org/10.1145/3303772.3303775

Counting clicks is not enough : validating a theorized model of engagement in learning analytics. / Fincham, Ed; Joksimović, Srećko; Whitelock-Wainwright, Alexander; Van Staalduinen, Jan Paul; Kovanović, Vitomir; Gašević, Dragan.

Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK'19): Learning Analytics to Promote Inclusion and Success. ed. / Christopher Brooks; Rebecca Ferguson; Ulrich Hoppe. New York NY USA : Association for Computing Machinery (ACM), 2019. p. 501-510 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - Counting clicks is not enough

T2 - validating a theorized model of engagement in learning analytics

AU - Fincham, Ed

AU - Joksimović, Srećko

AU - Whitelock-Wainwright, Alexander

AU - Van Staalduinen, Jan Paul

AU - Kovanović, Vitomir

AU - Gašević, Dragan

PY - 2019

Y1 - 2019

N2 - Student engagement is often considered an overarching construct in educational research and practice. Though frequently employed in the learning analytics literature, engagement has been subjected to a variety of interpretations and there is little consensus regarding the very definition of the construct. This raises grave concerns with regards to construct validity: namely, do these varied metrics measure the same thing? To address such concerns, this paper proposes, quantifies, and validates a model of engagement which is both grounded in the theoretical literature and described by common metrics drawn from the field of learning analytics. To identify a latent variable structure in our data we used exploratory factor analysis and validated the derived model on a separate sub-sample of our data using confirmatory factor analysis. To analyze the associations between our latent variables and student outcomes, a structural equation model was fitted, and the validity of this model across different course settings was assessed using MIMIC modeling. Across different domains, the broad consistency of our model with the theoretical literature suggest a mechanism that may be used to inform both interventions and course design.

AB - Student engagement is often considered an overarching construct in educational research and practice. Though frequently employed in the learning analytics literature, engagement has been subjected to a variety of interpretations and there is little consensus regarding the very definition of the construct. This raises grave concerns with regards to construct validity: namely, do these varied metrics measure the same thing? To address such concerns, this paper proposes, quantifies, and validates a model of engagement which is both grounded in the theoretical literature and described by common metrics drawn from the field of learning analytics. To identify a latent variable structure in our data we used exploratory factor analysis and validated the derived model on a separate sub-sample of our data using confirmatory factor analysis. To analyze the associations between our latent variables and student outcomes, a structural equation model was fitted, and the validity of this model across different course settings was assessed using MIMIC modeling. Across different domains, the broad consistency of our model with the theoretical literature suggest a mechanism that may be used to inform both interventions and course design.

KW - Engagement

KW - Factor analysis

KW - Measurement invariance

KW - MOOCs

KW - Structural equation modeling

UR - http://www.scopus.com/inward/record.url?scp=85062780578&partnerID=8YFLogxK

U2 - 10.1145/3303772.3303775

DO - 10.1145/3303772.3303775

M3 - Conference Paper

T3 - ACM International Conference Proceeding Series

SP - 501

EP - 510

BT - Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK'19)

A2 - Brooks, Christopher

A2 - Ferguson, Rebecca

A2 - Hoppe, Ulrich

PB - Association for Computing Machinery (ACM)

CY - New York NY USA

ER -

Fincham E, Joksimović S, Whitelock-Wainwright A, Van Staalduinen JP, Kovanović V, Gašević D. Counting clicks is not enough: validating a theorized model of engagement in learning analytics. In Brooks C, Ferguson R, Hoppe U, editors, Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK'19): Learning Analytics to Promote Inclusion and Success. New York NY USA: Association for Computing Machinery (ACM). 2019. p. 501-510. (ACM International Conference Proceeding Series). https://doi.org/10.1145/3303772.3303775