Developing and Evaluating an English Language Proficiency Test Using Generalizability Theory: A One-Facet Design

Abstract

The present study aimed to develop and evaluate an English language proficiency test using Generalizability Theory (G-Theory) and a one-facet person-by-item (p × i) design. The study addressed the need for a systematic examination of the sources of variability underlying English proficiency scores and the identification of an efficient test length. Sixty Iranian English language learners participated in the study and completed a researcher-developed 30-item multiple-choice English language proficiency test assessing vocabulary, grammar, and reading comprehension. The test was administered under standardized conditions, and participants’ responses were analyzed using an analysis of variance within a G-Theory framework. In the Generalizability Study (G-study), persons were treated as the objects of measurement and items as the single measurement facet. The analysis estimated the variance components associated with persons, items, and the person-by-item/residual component. The findings indicated that the person component accounted for a substantial proportion of the observed score variability (39.95%), whereas item-related variance was very small (0.13%). The person-by-item/residual component represented the largest proportion of variance (59.92%), suggesting that learners’ performance varied across items and that measurement error remained an important source of score variability. The 30-item test produced a high Generalizability coefficient (G = .952), indicating strong score dependability for relative decisions concerning learners’ English proficiency. A Decision Study (D-study) further examined alternative test lengths of 10, 15, 20, 25, and 30 items. The results showed that the G-coefficient increased systematically from .870 for 10 items to .952 for 30 items, while relative error variance decreased as the number of items increased. Although the 30-item test yielded the highest dependability, the 20- and 25-item versions also demonstrated high levels of score dependability, suggesting a practical balance between measurement precision and testing efficiency. The study concludes that G-Theory provides a useful framework for identifying sources of score variability and making evidence-based decisions about English proficiency test design and length.