Investigating Factors Associated With Gains in TOEFL ITP® Scores at a University in Japan

Abstract

This report presents the outcomes of a project funded by the TOEFL® English-Language Researcher/Practitioner Grant Program. The project examined how much first-year students in a Japanese university English program gained on the TOEFL ITP® test over the course of one academic year and which learner characteristics were associated with those gains. A cohort of 239 students in an intensive liberal arts English curriculum took the TOEFL ITP test three times. The students also completed measures of motivation, resilience, self-regulated learning (SRL), engagement (measured as peer support), and critical thinking, together with an end-of-year survey that asked whether the test reflected their English proficiency. Mean total scores increased from 426.2 to 455.2. The mean paired gain from Time 1 to Time 3 was 27.9 points (d_z = 0.85). However, the design included no comparison group, so the observed gain cannot be attributed to instruction alone. Critical-thinking performance was represented by a linked Rasch person measure that was constructed across three test forms. Critical thinking was the only focal learner variable that had a statistically significant unique association with end-of-year proficiency in regressions that controlled for Time 1 proficiency and form group (β = .358; the equivalent coefficient on the gain metric was .418). It was also associated with all three section scores (β = .239 to .352). No psychosocial composite had a significant unique association with end-of-year performance, although motivation and resilience were correlated with SRL (r = .40 and .42). Most learner measures were administered after the final test; therefore, the associations are concurrent or retrospective, and no causal interpretation is warranted. In the perception survey, 54% of students rated TOEFL ITP as “more or less” reflective of their English proficiency. The open-ended comments centered on the absence of speaking and writing from the regular score. The report discusses the implications for score interpretation, program evaluation, and future TOEFL ITP research.

Suggested citation: Park, S., & Tetsuya, F. (in press). Investigating factors associated with gains in TOEFL ITP® scores at a university in Japan (Research Memorandum). ETS.