How difficult are exams? A framework for assessing the complexity of introductory programming exams

Judy Sheard, Simon, Angela Carbone, Donald Chinn, Tony Clear, Malcolm Corney, Daryl D'Souza, Joel Fenwick, James Harland, Mikko-Jussi Laakso, Donna Teague

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

12 Citations (Scopus)

Abstract

Student performance on examinations is influenced by the level of difficulty of the questions. It seems reasonable to propose therefore that assessment of the difficulty of exam questions could be used to gauge the level of skills and knowledge expected at the end of a course. This paper reports the results of a study investigating the difficulty of exam questions using a subjective assessment of difficulty and a purpose-built exam question complexity classification scheme. The scheme, devised for exams in introductory programming courses, assesses the complexity of each question using six measures: external domain references, explicitness, linguistic complexity, conceptual complexity, length of code involved in the question and/or answer, and intellectual complexity (Bloom level). We apply the scheme to 20 introductory programming exam papers from five countries, and find substantial variation across the exams for all measures. Most exams include a mix of questions of low, medium, and high difficulty, although seven of the 20 have no questions of high difficulty. All of the complexity measures correlate with assessment of difficulty, indicating that the difficulty of an exam question relates to each of these more specific measures. We discuss the implications of these findings for the development of measures to assess learning standards in programming courses.

Original languageEnglish
Title of host publicationProceedings of the Fifteenth Australasian Computing Education Conference (ACE 2013)
Subtitle of host publicationAdelaide, Australia, 29 January - 1 February 2013
EditorsAngela Carbone, Jacqueline Whalley
Place of PublicationSydney, New South Wales
PublisherAustralian Computer Society Inc
Pages145 - 154
Number of pages10
Volume136
ISBN (Electronic)9781921770210
ISBN (Print)9781921770210
Publication statusPublished - 29 Jan 2013
EventAustralasian Computing Education Conference 2013 - Adelaide, Australia
Duration: 29 Jan 20131 Feb 2013
Conference number: 15th
https://dl.acm.org/doi/proceedings/10.5555/2667199 (Proceedings)

Publication series

NameConferences in Research and Practice in Information Technology (CRPIT)
PublisherAustralian Computer Society Inc.
Volume136
ISSN (Print)1445-1336

Conference

ConferenceAustralasian Computing Education Conference 2013
Abbreviated titleACE 2013
CountryAustralia
CityAdelaide
Period29/01/131/02/13
Internet address

Keywords

  • Standards
  • Quality
  • Examination papers
  • CS1
  • Introductory programming
  • Assessment
  • Question complexity
  • Question difficulty

Cite this