Validity theory (VT) and modern test theory (MTT) in assessment measurement and design

Blyden, Susan (2021). Validity theory (VT) and modern test theory (MTT) in assessment measurement and design. University of Birmingham. Ed.D.

Preview

BlydenEdD2021_Redacted.pdf
Text - Redacted Version
Available under License All rights reserved.
Download (13MB) | Preview

Abstract

Background: Healthcare education use assessment results (test scores) to make inferences about students’ knowledge and ability. The NMP curriculum, therefore, assumes a standard of proficiency achievement in successive student cohorts. Therefore, these assessments are high-stakes and must be valid as they may have implications for patients’ safety.

Aims: This study aimed to apply Kane’s (2006) validity theory to evidence whether the assessments capture the proficiency it purports to assess and modern test theory to measure the appropriateness of the score interpretation.

Method: Theory-Driven Thematic Analysis (TDTA) was used to code and content analyse the reflections; Ryle (1949), Mezirow (1981), Benner (2001) and Bloom’s (1956) theories instructed the content validity, quality and evidenced mastery of the NMP subject and theory domains. The Structured Clinical Examination (SCE) criterion was analysed using the Rasch one-parameter logistic model (1-PL) to measure relationships between the test item criteria, the student ability and test-items parameters capacity to discriminate among students with varying ability.

Results: Representation of Ryle’s, Mezirow’s, Benner’s, and Bloom’s theories in the reflections were at the lower levels of achievement. The SCEs, criterion analyses showed the test item favoured students with limited ability, indicating an easy test; overall, suggesting the assessments’ validity is compromised.

Conclusion: Applying validity and measurement theory to assessed work could elucidate the meaning of assessment scores and ensure that assessments are appropriate to the examinees’ level of ability.

Type of Work:

Thesis (Doctorates > Ed.D.)

Award Type:

Doctorates > Ed.D.

Supervisor(s):