• Facet Variability in the Light of Rater Training in Measuring Oral Performance: A Multifaceted Rasch Analysis

    Author(s):
    Houman Bijani, Salim Said Bani Orabah (see profile)
    Date:
    2023
    Group(s):
    Education and Pedagogy, General Education, Linguistics, Open Educational Resources, Public Humanities
    Item Type:
    Article
    Permanent URL:
    https://doi.org/10.17613/wxd5-w440
    Abstract:
    Due to subjectivity in oral assessment, much concentration has been put on obtaining a satisfactory measure of consistency among raters. However, obtaining consistency might not result in valid decisions. One matter that is at the core of both reliability and validity in oral performance is rater training. Recently, the Multifaceted Rasch Measurement (MFRM) has been adopted to address the problem of rater bias and inconsistency; however, no research has incorporated the facets of test takers’ ability, raters’ severity, task difficulty, group expertise, scale criterion category, and test version together in a piece of research along with their two-sided impacts. Moreover, little research has investigated how long rater training effects endure. Consequently, this study explored the influence of the training program and feedback by having 20 raters score the oral production, as measured by the CEP (Community English Program) test, produced by 300 test takers in three phases, i.e., before, immediately after, and long after the training program. The results indicated that training can lead to higher degrees of interrater reliability and decrease in measures of severity/leniency and biasedness. However, it did not lead the raters into total unanimity, except for making them more self-consistent. Although rater training might result in higher internal consistency among raters, it cannot eradicate individual differences. That is, experienced raters, due to their idiosyncratic characteristics, did not benefit as much as the inexperienced ones. This study also showed that the outcome of training might not endure in long terms after training; thus, it requires ongoing training, letting raters regain consistency.
    Metadata:
    xml
    Published as:
    Journal article     Show details
    Pub. DOI:
    https://doi.org/10.22054/ilt.2023.63589.634
    Pub. Date:
    4/6/2023
    Journal:
    Issues in Language Teaching
    Volume:
    11
    Issue:
    2
    Page Range:
    255 - 290
    Status:
    Published
    Last Updated:
    6 months ago
    License:
    All Rights Reserved

    Downloads

    Item Name: pdf ilt_volume-11_issue-2_pages-255-290.pdf
      Download View in browser
    Activity: Downloads: 12

    Back to Deposits