The ETS Research Report Series features peer-reviewed articles based on measurement and education research conducted by the scientists at ETS. Our reports have been available to teachers, policy makers, and the general public since our first volume was published in 1948. Topics range from English language learning and mathematics skills to teacher preparedness and workforce development to assessments and artificial intelligence. Articles are published in three sections: reports, memoranda, and notes. Issues prior to 2025 are available on the Wiley Online Library. Our staff includes Executive Editor Daniel F. McCaffrey, Lord Chair in Measurement and Statistics, and more than a dozen associate editors. Our full backlist is available at the Wiley Online Library.
TOEIC® Link™ Assessments Technical Manual
This technical manual provides a comprehensive overview of the TOEIC® Link™ assessments, offering detailed insights into their purpose, design, and intended users. The manual begins with an introduction that outlines the assessment objectives and target audience. Subsequent sections delve into the specific constructs and tasks of the four TOEIC Link assessments: Listening, Reading, Speaking, and Writing. The remaining sections examine the design and development processes for both the listening and reading, as well as the speaking and writing assessments, highlighting methodologies used to ensure validity and reliability. Together, these sections present a thorough guide for test users and stakeholders seeking to understand the TOEIC Link assessments and their application in measuring English language proficiency. Designed as a living document, this manual will be updated as the test’s design, administration, scoring, and evidence of measurement quality (including reliability, validity, and fairness) evolve, along with its intended uses. Suggested citation: Cid, J., Schmidgall, J., & Park, E. (2026). TOEIC® Link™ assessments technical manual (Research Report No. RR-26-04). ETS. https://doi.org/10.64634/7e2pzg04
Copyright (c) 2026 Educational Testing Service https://creativecommons.org/licenses/by-nc-nd/4.0
Beyond Score Correlations: A Content Comparison of IELTS Academic and TOEFL iBT® Tests in the Context of a Score Concordance Study
Score concordance tables are widely used by higher education institutions to compare scores from different English language proficiency tests, yet their validity depends on the extent to which the tests measure comparable constructs. This study examines the content comparability of the International English Language Testing System (IELTS) Academic and TOEFL iBT® tests in the context of a recent co-sponsored score concordance study. Moving beyond score correlations, the analysis compares test content across the four language skills—reading, listening, writing, and speaking—using published research, official test documentation, and publicly available sample materials. The comparison is framed by the validity frameworks adopted by each test provider and focuses on task characteristics, linguistic demands, response formats, and scoring criteria. Results indicate substantial overlap in the constructs assessed by both tests, particularly in reading and writing, where tasks target similar academic language skills and are evaluated using comparable criteria. More pronounced differences emerge in listening and speaking, with TOEFL iBT placing greater emphasis on academic content, integrated skills, and pragmatic inference, while IELTS includes more general listening contexts and examiner-mediated interaction in speaking. Despite these differences, both tests provide multiple opportunities for test takers to engage with extended discourse and demonstrate receptive and productive language abilities. Overall, the findings support the use of score concordance tables between IELTS Academic and TOEFL iBT, while emphasizing the need for cautious interpretation and recognition that scores should not be treated as fully interchangeable. Suggested citation: Cushing, S. T. (2026). Beyond score correlations: A content comparison of IELTS Academic and TOEFL iBT® tests in the context of a score concordance study (TOEFL Research Report No. RR-107). ETS. https://doi.org/10.64634/qercg225
The Algorithm at the Gate: Exploring Ethical Tensions in Graduate Admissions
As generative artificial intelligence (AI) rapidly reshapes how information is created, shared, and evaluated, graduate admissions professionals are confronting new questions about its role in recruitment and selection. This report draws upon ETS’s February 2025 pulse survey of NAGAP members to explore how graduate enrollment management (GEM) professionals are navigating this evolving landscape. We present their perspectives on the acceptable use of AI in graduate admissions—both by institutions and by applicants—and highlight emerging norms and tensions. The report concludes with a discussion on the implications of these findings for GEM policies and practice.
Toward an Automatic Method for Generating Topical Vocabulary Test Forms for Specific Reading Passages
Background knowledge is typically needed for successful comprehension of topical and domain-specific reading passages, such as in the STEM domains. However, there are few automated measures of student knowledge that can be readily deployed and scored in time to make predictions on whether a given student will likely be able to understand a specific content-area text. In this research report, we present our effort in developing the K-tool, an automated system for generating topical vocabulary tests that measure students’ background knowledge related to a specific text. The system automatically detects the topic of a given text and produces topical vocabulary items based on their relationship with the topic. This information is used to automatically generate background knowledge forms that contain words that are highly related to the topic and words that share similar features but do not share high associations to the topic. Prior research has indicated that performance on such tasks can help determine whether a student is likely to understand a particular text based on their knowledge state. The described system is intended for use with middle and high school student populations of native speakers of English. It is designed to handle single reading passages and is not dependent on any corpus or text collection. In this report, we describe the system architecture and present an initial evaluation of the system outputs. Suggested citation: Flor, M., Wang, Z., Deane, P., & O’Reilly, T. (2025). Toward an automatic method for generating topical vocabulary test forms for specific reading passages (Research Report No. RR-26-02). ETS.
A Preliminary Research and Evaluation Agenda for Personalized Assessment in the Service of Equity
This research memorandum describes a preliminary research and evaluation agenda for personalized assessments, where such assessments are intended to be attuned to the social, cultural, and other relevant characteristics of individuals and the contexts from which they come. The agenda targets the full range of assessment uses—school accountability, national and international assessment, admissions, certification and licensure, and instructional planning. The purposes of the agenda are to guide the theoretical and empirical research and development needed to create personalized assessments and to suggest a means for judging the effectiveness of those instruments. Suggested citation: Bennett, R. E., Sparks, J. R., Arslan, B., Lehman, B., Sinharay, S., & Zapata-Rivera, D. (2025). A preliminary research and evaluation agenda for personalized assessment in the service of equity (Research Memorandum No. RM-26-01). ETS. https://doi.org/10.64634/wjv5e895
Keep Calm and Carry On: A Conceptual Framework for Perseverance
Perseverance is a highly valued durable skill—a set of constructs also referred to as socioemotional skills, noncognitive constructs, transferrable skills, and 21st century skills, among other terms—as it describes one’s ability to manage challenges and obstacles in academic, workplace, and other life areas. As part of a new critical skills taxonomy designed to include academic and employability skills beyond traditional academic content knowledge, our novel conceptual framework of perseverance addresses the ambiguities of previous perseverance definitions. Specifically, our perseverance definition includes two subskills: emotional resilience and effortful persistence. We approach conceptualization with an eye towards relevance for youth settings (e.g., pre-kindergarten–Grade 12). We also compare this conceptual framework to previous definitions and distinguish it from overlapping constructs. We then describe each subskill’s relevance to outcomes across various settings. Additional implications of our conceptual framework, including measurement considerations and malleability, are discussed. Suggested citation: Williams, K. M., Ling, G., Sotelo, J., Brantley, W., Wang, Y., & Kinsey, D. (2026). Keep calm and carry on: A conceptual framework for perseverance (Research Report No. RR-26-01). ETS. https://doi.org/10.64634/5qd2rx29
Using Ordinal Rescore Measures to Monitor Rater Drift
When constructed response items are used on more than one occasion, a natural concern is whether the scoring is consistent (e.g., not more lenient or strict) across the occasions. It is common to conduct trend scoring, in which a set of Occasion A responses are rescored at Occasion B. The responses are usually selected according to some rescore design, such as being balanced (with an equal number from each score category), proportional to the distribution of Occasion A scores, or a mixed version of these two designs. Recent work has demonstrated that treating the two-way table as if it arose from multinomial sampling is incorrect and can yield seriously biased estimates of whether the scores are lower or higher at Occasion B. The present study builds on these results by incorporating ordinal measures of change. It contrasts the usual trend analysis with an alternative analysis that explicitly conditions on the rescore design and finds only the latter to be effective. Omnibus measures based on combining the individual t-tests or d-statistics are examined. Measures were somewhat conservative in Type I error control and had good power to detect drift. Omnibus measures based on t-tests had marginally higher power, having higher correct detection rates than those based on the d-statistic in 1%–8% of the cases. The difference between the best versions (E weighted, which is based on t-tests, vs. D weighted, which is based on d-statistics) was only 1.8%. Suggested citation: Donoghue, J. R., & Sgammato, A. (2025). Using ordinal rescore measures to monitor rater drift (Research Report No. RR-25-15). ETS.
Copyright (c) 2025 Educational Testing Service https://creativecommons.org/licenses/by-nc-nd/4.0
Graduate Admissions After Affirmative Action: Results From the ETS/NAGAP 2024 Admissions Survey
On the tail of changes to testing policies and other graduate admissions practices through the COVID-19 era, 2023 saw a further major change in the admissions landscape through the U.S. Supreme Court’s striking down of race-based affirmative action in admissions processes through the Students for Fair Admissions (SFFA) cases. The ETS Research Institute and NAGAP, the Association for Graduate Enrollment Management, partnered on research to understand how this ruling is affecting interest holders and, in particular, how it has affected practices of holistic admissions, which have often been defined with respect to the explicit goal of promoting diversity (see, e.g., Glazer et al., 2014). This research note focuses on the viewpoints of graduate admissions officers and department leaders surveyed in 2024 on the effects they expected to see from the U.S. Supreme Court’s decision on affirmative action in admissions processes and on the reactions that they were seeing at their own institutions. Suggested citation: Gooch, R., & Haviland, S. (2025). Graduate admissions after affirmative action: Results from the 2024 ETS/NAGAP Survey (Research Notes). ETS. https://doi.org/10.64634/5s9fnm23
Preparing K–12 Students With AI Literacy: Proposed Framework, Progression, and Task Design Principles
This paper presents a conceptual framework for AI literacy, a hypothesized learning progression, and assessment design principles for advancing AI literacy among K–12 learners. Recognizing the importance of technical competencies alongside ethical awareness, the framework integrates foundational knowledge, societal implications, and practical applications of AI. Key competencies include ethical decision-making, AI-powered collaboration, and critical evaluation of AI outputs. Developed through an evidence-centered design (ECD) process involving a review of existing literature and frameworks, the proposed AI literacy framework and progression maps a hypothesized trajectory of students’ skill development, providing a structured pathway for improvement with behavior indicators connected to core AI literacy subskills. In this way, the framework and progression may offer educators a roadmap to apply scaffolded and differentiated teaching strategies that actively foster learners’ skill acquisition. To further support connections between assessment and instruction, we introduce three design principles for task design: ensuring relevance to learners, minimizing barriers to resource access, and providing opportunities for skill advancement. These design principles may guide the creation of activities that evaluate and enhance students’ AI literacy. By aligning scaffolded assessments and learning activities with the progression, this framework bridges instruction, assessment, and students’ skill development. It ultimately may be used to support students in developing skills to critically and ethically engage with AI technologies, preparing them to navigate the digital landscape by fostering inclusive instruction that deepens students’ understanding of AI concepts. Chakraburty, S., Ober, T. M., & Liu, L. (2025). Preparing K–12 students with AI literacy: Proposed framework, progression, and task design principles (Research Report No. RR-25-14). ETS. https://doi.org/10.64634/46jn1p41
An Evaluation of Item Fit Based on Generalized Residual Item Response Functions
Evaluation of item ft for item response theory (IRT) models often involves a comparison of the observed and expected item response functions (IRFs). Several statistics have been suggested for evaluating item ft based on the discrepancy between IRFs, but the asymptotic distributions of the statistics under the null hypothesis are often not well established. Haberman et al. developed a method for evaluating the ft of IRFs based on generalized residuals. These residuals are functions of the latent proficiency variable in the IRT model and follow the standard normal distribution asymptotically. We develop a method to summarize these generalized residuals into a single summary statistic for each item and evaluate its asymptotic distribution. Kondratek suggested a similar Wald-type statistic, but without accounting for the uncertainty in the estimation of the item parameters. Our method combines the work of Haberman and Kondratek, resulting in a single ft statistic per item while accounting for estimation error. A series of simulations was carried out to investigate the performance of our statistic and compare it to several popular item ft statistics. Our method resulted in similar Type I errors as Kondratek’s statistic, with slightly better results in the case of small samples. Furthermore, the recovery was consistent across different levels of item difficulty, and power of the new item ft statistic was relatively low, except for problematic individual items, but this result was found with two competing statistics as well. Suggested citation: Liao, X., van Rijn, P., & Sinharay, S. (2025). An evaluation of item fit based on generalized residual item response functions (Research Report No. RR-25-13). ETS. https://doi.org/10.64634/b68vz316
This technical manual describes the purpose and intended uses of the TOEFL iBT test, its target test-taker population, and relevant language use domains. The test design and scoring procedures are presented first, followed by a research agenda intended to support the interpretation and use of test scores. Given the updates to the test starting January 2026, this technical manual is intended to serve as an overview and rationale for the test design as well as a reference point for informing investigations of validity evidence to support the intended test uses over time. Designed as a living document, this manual will be updated as the test's design, administration, scoring, and evidence of measurement quality (including reliability, validity, and fairness) evolve, along with its intended uses. Suggested citation: Manna, V. F., Li, S., Papageorgiou, S., & Gu, L. (2025). TOEFL iBT® technical manual (TOEFL Research Report No. RR-106). ETS. https://doi.org/10.64634/eje8f497
Pathways to Graduate School: 5. A Data Overview of U.S. Prospective Graduate Students by Black Students and Gender
One understudied segment of the domestic talent pool is the Black or African American individuals who are U.S. citizens and aspire to earn a graduate degree. Gender differences within this talent pool are specifically understudied. The 101,039 individuals who are U.S. citizens, who provided gender and race/ethnicity information, and who had scores on all three GRE® measures are the subjects for this descriptive study, one of a series of five such reports. GRE data from July 1, 2016, to June 30, 2021, supplemented with U.S. Department of Education and U.S. Census data, are analyzed. These individuals, referred to as prospective graduate students (PGS), are examined through six core questions: (a) Who were the PGS? (b) Where did they reside? (c) What were their education and work experiences? (d) What were their undergraduate experiences? (e) What were their plans for graduate study? and (f) What were their emerging graduate school choice sets? Key findings include the following: (a) 34% were 22 years of age or younger; (b) 37% were currently enrolled in college; (c) 75% attended an institution in their home state, with women doing so slightly more than men; (d) 46% were first-generation college students; (e) nearly 60% were Pell Grant–eligible; (f) as undergraduates, women favored life sciences, while men leaned toward physical sciences and engineering; (g) women were slightly more interested in master’s degrees; and (h) 76% included at least one in-state institution in their graduate school choice sets. The report concludes with recommendations for future research and practical applications, particularly in the graduate school application process. Suggested citation: Millett, C. M. (2025). Pathways to graduate school: 5. A data overview of U.S. prospective graduate students by Black students and gender (Research Report No. GRE-25-05). ETS.
Pathways to Graduate School: 4. A Data Overview of U.S. Prospective Graduate Students by Pell Grant Eligibility and Gender
Today, the Federal Pell Grant program is ubiquitous in conversations about how to pay for undergraduate education. Since 1982, it has supported more than 212 million students, making it a cornerstone of higher education access in the United States. However, what happens next for Pell Grant recipients who earn bachelor’s degrees—especially when pursuing graduate study—remains largely unexplored. This knowledge gap leaves questions unanswered about how gender and financial need shape the path beyond college. The 673,573 individuals who are U.S. citizens, who provided gender and their undergraduate Pell Grant eligibility information, and who had scores on all three GRE® General Test measures are the subjects for this descriptive study, one of a series of five such reports. GRE General Test data from July 1, 2016, to June 30, 2021, supplemented with U.S. Department of Education and U.S. Census data, are analyzed. These individuals, referred to as prospective graduate students (PGS), are examined through six core questions: (a) Who were the PGS? (b) Where did they reside? (c) What were their education and work experiences? (d) What were their undergraduate experiences? (e) What were their plans for graduate study? and (f) What were their emerging graduate school choice sets? Key findings include the following: (a) 50% of Pell-eligible PGS were White, and 75% of non-Pell-eligible PGS were White; (b) fewer Pell-eligible PGS were enrolled in college; (c) more Pell-eligible PGS were enrolled at in-state institutions; (d) substantially more Pell-eligible PGS were first-generation college students compared to their non-Pell-eligible peers; (e) more men than women in both Pell eligibility profiles majored in the physical sciences and engineering than in the life sciences; (f) more than 90% of PGS of both Pell eligibility profiles earned a 3.0 or higher grade point average in their undergraduate majors; and (g) most PGS were considering at least one graduate school in their states of residence. The report concludes with recommendations for future research and practical applications, particularly in the graduate school application process. Suggested citation: Millett, C. M. (2025). Pathways to graduate school: 4. A data overview of U.S. prospective graduate students by Pell Grant eligibility and gender (Research Report No. GRE-25-04). ETS.
Pathways to Graduate School: 3. A Data Overview of U.S. Prospective Graduate Students by Parental Education and Gender
Having parents with higher educational attainment is associated with many positive academic outcomes for students (e.g., college enrollment, degree attainment). This study aims to raise awareness and promote understanding of the characteristics, aspirations, and graduate degree intentions of students who are considering graduate school, particularly through the lens of parental education and gender. Three distinct groups are profiled: no parent bachelor of arts (BA), one parent BA, and one parent BA+. The 980,903 individuals who are U.S. citizens, provided gender and parent educational attainment information, and had scores on all three GRE® General Test measures are the subjects for this descriptive study, one of a series of five such reports. GRE General Test data from July 1, 2016, to June 30, 2021, supplemented with U.S. Department of Education and U.S. Census data, are analyzed. These individuals, referred to as prospective graduate students (PGS), are examined through six core questions: (a) Who were the PGS? (b) Where did they reside? (c) What were their education and work experiences? (d) What were their undergraduate experiences? (e) What were their plans for graduate study? and (f) What were their emerging graduate school choice sets? Key findings include the following: (a) The percentage of White PGS increased as parental education increased; (b) more than 50% of PGS from all three parental education groups live in 10 states; (c) Federal Pell Grant eligibility decreased as parental education increased; (d) approximately half of PGS in each parental education group majored in a science, technology, engineering, or mathematics field, with notable gender differences; (e) more than 90% of PGS across all parental education groups reported a B or higher undergraduate major grade point average; (f) a master’s degree was the most common degree objective among all three parental education profiles; (g) most PGS were considering attending at least one graduate school in their home state; and (h) 41% of PGS included at least one minority-serving institution in their graduate school choice sets. The report concludes with recommendations for future research and practical applications, particularly in the graduate school application process. Suggested citation: Millett, C. M. (2025). Pathways to graduate school: 3. A data overview of U.S. prospective graduate students by parental education and gender (Research Report No. GRE-25-03). ETS.
Pathways to Graduate School: 2. Data Overview of U.S. Prospective Graduate Students by Hispanic Subgroup and Gender
While discussions around increasing diversity often focus on attracting a broad range of students, the unique needs and challenges of specific groups—such as Hispanic graduate school aspirants—are frequently overlooked. This oversight is problematic given the growing demographic significance of Hispanic individuals in the United States and their underrepresentation in the highest levels of academia. Hispanic graduate students represent a small but vital part of America’s future intellectual capital. Even less frequently explored is the rich heterogeneity of Hispanic graduate school aspirations, with a particular focus on Hispanic subgroup differences and gender. The 120,445 individuals who are U.S. citizens, who provided gender and race/ethnicity information, and had scores on all three GRE® General Test measures are the subjects for this descriptive study, one of a series of five such reports. GRE General Test data from July 1, 2016, to June 30, 2021, supplemented with U.S. Department of Education and U.S. Census data, are analyzed. These individuals, referred to as prospective graduate students (PGS), are examined through six core questions: (a) Who were the PGS? (b) Where did they reside? (c) What were their education and work experiences? (d) What were their undergraduate experiences? (e) What were their plans for graduate study? and (f) What were their emerging graduate school choice sets? Key findings include the following: (a) For Mexican, Puerto Rican, and other Hispanic PGS, roughly 68% are 25 years of age or younger, with many being under 22 years of age; (b) the three Hispanic PGS groups share six states where they reside in high numbers, but each subgroup has at least one unique state where they are present in significant numbers; (c) across the three groups, more than two-thirds attended a public or private baccalaureate institution in their states of residence; (d) parental education varied across three groups; (e) in all three Hispanic subgroups, more men than women majored in physical sciences and engineering, while more women than men majored in the life sciences; (f) a master’s degree was the most common graduate degree objective, with greater interest for women than for men; and (g) there were considerable differences across the three groups regarding including at least one minority-serving institution in their graduate school choice sets. The report concludes with recommendations for future research and practical applications, particularly in the graduate school application process. Suggested citation: Millett, C. M. (2025). Pathways to graduate school: 2. A data overview of U.S. prospective graduate students by Hispanic subgroup and gender (Research Report No. GRE-25-02). ETS.
Pathways to Graduate School: 1. A Data Overview of U.S. Women Prospective Graduate Students—Profiles of Nine Racial Groups
Understanding the diverse pathways women take toward graduate education is crucial but often overlooked. When these journeys are presented, they are typically compared to men’s experiences, which can obscure the unique challenges and opportunities women face. This descriptive research study, one of five reports, aims to provide a comprehensive profile of U.S. women who may pursue graduate study, disaggregated by race/ethnicity. Nine distinct profiles of women, based on race/ethnicity, are presented. The 698,298 women who provided gender and race/ethnicity information and had scores for all three GRE® test measures are the subjects. GRE General Test data from July 1, 2016, to June 30, 2021, supplemented with U.S. Department of Education and U.S. Census data, are analyzed. These women, referred to as prospective graduate students (PGS), are examined through six core questions: (a) Who were the women PGS? (b) Where did they reside? (c) What were their education and work experiences? (d) What were their undergraduate experiences? (e) What were their plans for graduate study? and (f) What were their emerging graduate school choice sets? Key findings include the following: (a) The majority of women PGS across the nine profiles lived in 10 states, however, the composition of these 10 states varied by group; (b) two out of five women PGS were enrolled in college, with considerable variation across the nine racial profiles; (c) across the profiles, there was a 40 percentage point difference in having no parent with a bachelor’s degree, ranging from 68% of Mexican women to 28% of White women; (d) 33% of women PGS reported being eligible for a Federal Pell Grant, ranging from 24% of White women to 60% of Black women; (e) women across the nine profiles were comparable in majoring in the physical sciences or engineering as undergraduates but differed substantially in majoring in life sciences; and (f) 32% of all women PGS considered a regional comprehensive university in their state of residence for graduate school, with notable variation, ranging from 42% of Mexican women to 28% of Asian women. Suggested citation: Millett, C. M. (2025). Pathways to graduate school: 1. A data overview of U.S. women prospective graduate students—Profiles of nine racial groups (Research Report No. GRE-25-01). ETS.
Navigating the Digital Horizon: A Proposed Framework and Strategies for Assessing Digital Literacy
Digital literacy skills are essential for success in today’s social, academic, and professional environments. This paper introduces a digital literacy framework, outlining key subskills and skill indicators for high school students, grounded in prior research and K–12 educational standards. We examine the implementation of digital literacy assessments in K–12 education and offer recommendations for their integration into educational systems. Our goal is to support the thoughtful development, assessment, and feedback processes necessary to foster students’ digital literacy skills. Suggested citation: Liu, L., Courey, K. A., Kinsey, D., Ober, T. M., & Johnson, D. G. (2025). Navigating the digital horizon: A proposed framework and strategies for assessing digital literacy (Research Report No. RR-25-06). ETS.
Communication as a Future Ready Skill: A Proposed Framework and Strategies for Assessment
Effective communication skills are essential for success in both academic and professional contexts. This concept paper presents a novel framework designed to operationally define and support the development of assessments of communication skills with a specific emphasis on K–12 settings. Through discussions on important considerations for the conceptualization, development, implementation, and use of assessments of communication skills, this paper aims to equip educators and interest holders with insights to foster effective assessment practices of communication skills. Ultimately, we hope this framework may support the identification, development, and assessment of communication skills among school-age learners as we strive to empower students to navigate and excel in the future. Suggested citation: Ober, T. M., Johnson, D. G., Liu, L., Kinsey, D., & Courey, K. A. (2025). Communication as a future ready skill: A proposed framework and strategies for assessments (Research Report No. RR-25-05). ETS.
Improving the Quality of Numeracy Skills: Progressing from Level 2 to Level 4 on the PIAAC Cycle 2 Proficiency: An ETS Return on Investment Study: Phase 1
This research memorandum highlights the collaborative efforts of the numeracy panel for the ETS Return on Investment (ROI) Study, Phase 1, comprising David (Dave) Tout, Kees Hoogland, and Javier Díez-Palomar, in developing their research memorandum, Improving the Quality of Numeracy Skills: Progressing from Level 2 to Level 4 on the PIAAC Cycle 2 Proficiency Scale. This memorandum captures the expert opinions of the panel sessions. Suggested citation: Tout, D., Hoogland, K., & Díez-Palomar, J. (2025). Improving the quality of numeracy skills: Progressing from Level 2 to Level 4 on the PIAAC Cycle 2 proficiency scale: An ETS return on investment study, Phase 1 (Research Memorandum No. RM-25-02). ETS.
Literacy Skills at and Around Level 2 of the PIAAC Cycle 2 Proficiency Scale An ETS Return on Investment Study: Phase 1
This research memorandum highlights the collaborative efforts of the literacy panel for the ETS Return on Investment (ROI) Study, Phase 1, comprising Jean-François Rouet, M. Anne Britt & Tobias Richter, in developing their research memorandum, Literacy Skills at and Around Level 2 of the PIAAC Cycle 2 Proficiency Scale. This memorandum captures the expert opinions from the panel sessions. Suggested citation: Rouet, J.-F., Britt, A., & Richter, T. (2025). Literacy skills at and around Level 2 of the PIAAC Cycle 2 proficiency scale: An ETS return on investment study, Phase 1 (Research Memorandum No. RM-25-01). ETS.
Copyright (c) 2025 ETS https://creativecommons.org/licenses/by-nc-nd/4.0
Level Up: Raising the Skills of Adults in the United States and Other Countries
In an era of globally competitive and technology-driven societies, we are seeing a growing interest in developing a better understanding of the types of skills adults need to succeed both in the workplace and in everyday life. One large-scale comparative survey designed to inform that understanding is the OECD’s Program for the International Assessment of Adult Competencies (PIAAC), a survey of adults ages 16-65 conducted in over 30 countries. PIAAC was designed for, and with, participating countries and represents a wide range of knowledge and skills that are required across a variety of adult contexts. The PIAAC data make it clear that adults with lower-level literacy and numeracy skills experience less favorable social, educational and labor market outcomes when compared with their more highly skilled cohorts. While some of those adults possess key foundational reading and numeracy skills, they would benefit from enhancing their skills so that they can more readily navigate, critically analyze, and problem solve in today’s data-intensive, complex digital environments. This report describes a project in which two groups of experts – one in the domain of literacy and the other in numeracy – conducted analyses of the PIAAC data with the goal of defining the knowledge and skills associated with various levels of proficiency in those domains. The key purpose of this work is to form a foundation for the development of strategic interventions to improve adult literacy and numeracy skills. The argument made herein is that the insights gained from these expert analyses into the skills needed to transition to higher levels of literacy and numeracy can be leveraged to create a coherent learning and assessment system that could significantly enhance the literacy and numeracy skills of various adult populations. Suggested citation: Kirsch, I., Lennon, M. L., Sands, A., with Rouet, J.-F., Britt, A., Richter, T., Tout, D., Hoogland, K., and Diez-Palomar, J. (2025). Level up: Raising the skills of adults in the United States and other countries (Research Report No. RR-25-04). ETS.
Program for the International Assessment of Adult Competencies (PIAAC)
Responsible AI for Measurement and Learning: Principles and Practices
The rapid proliferation of artificial intelligence (AI) in educational measurement presents both transformative opportunities and complex ethical challenges. This paper articulates foundational principles for the responsible integration of AI in measurement and learning, drawing on established guidelines set forth by leading organizations such as NIST, OECD, UNESCO, the U.S. Department of Education, and others. We propose a principled framework encompassing fairness and bias mitigation, privacy and security, transparency, explainability, accountability, educational impact and integrity, and continuous improvement. Through the synthesis of current research, best practices, and cross-sector standards, we highlight practical measures to ensure that AI-driven assessment systems are equitable, valid, and reliable. Special emphasis is placed on the significance of representative data, ongoing bias analysis, secure-by-design development, and stakeholder involvement throughout the AI lifecycle. This approach is designed to foster trust, uphold educational values, and safeguard individual rights. By emphasizing ethical and sustainable practices, we advocate for a vision of AI as a driver of human development—supporting learners, educators, and society at large in the pursuit of educational and economic mobility. The principles and recommendations outlined here offer guidance not only for our organization but serve as a resource for the broader educational and measurement community, charting a course for responsible AI innovation that advances both the science and the practice of measurement in support of lifelong learning. Suggested citation: Johnson, M. S. (2025). Responsible AI for measurement and learning: Principles and practices (Research Report No. RR-25-03). ETS.
Aligning Scores of Language Proficiency Tests: A Score Concordance Study Between IELTS Academic and TOEFL iBT®
Scores of different language tests intended for similar purposes (e.g., admission to higher education) are used to determine candidates’ language proficiency and readiness for a chosen domain. To be fair to all students irrespective of the test they took, score requirements should be comparable. Score concordance tables provide an empirical basis for such comparability when good practice principles are met. We report on a score concordance project whose ambitious goal was to adhere to the good practice principles laid out in Knoch and Fan (2024). The providers of the two most widely-used English language proficiency tests for academic admissions purposes, IELTS Academic and TOEFL iBT®, collaborated to complete this project. Research teams representing both tests recruited 969 test takers who took the tests in a counterbalanced order. The study participants represented the major first language groups of both test-taking populations. Every effort was made to keep the interval between taking the two tests short to minimize any effect of changes in the test-takers’ language proficiency level. There were no self-reported score data, a major limitation of existing score concordance studies, as all score reports were verified by the test providers using their official score verification service. Equipercentile equating was conducted by a third party, independent from the test providers. We discuss the challenges in meeting several good practice principles and present the implications for building trustworthy score concordance tables to help stakeholders make informed decisions about language test acceptance. Suggested citation: Ikeda, N., Clark, T., Papageorgiou, S., Gu., L., Ohta, R., Blackhurst, A., & Bruce, E. (2025). Aligning scores of language proficiency tests: A score concordance study between IELTS Academic and TOEFL iBT® (TOEFL Research Report No. RR-105). ETS.
Testing Academic Language Proficiency: Comparing the TOEFL iBT® Test and the Duolingo English Test
This report provides an in-depth comparison of TOEFL iBT® and the Duolingo English Test (DET) in terms of the degree to which both tests assess academic language proficiency in listening, reading, writing, and speaking. The analysis is based on publicly available documentation on both tests, including sample test questions available on the test websites. For each skill area, the construct as defined by the test developer is discussed, followed by a review of the test content and the cognitive processes involved in completing items related to that skill. The results are evaluated against three propositions from the TOEFL® validity argument (Chapelle et al., 2008). The analysis suggests that although both tests provide evidence of general language proficiency, the input to test takers and the test items for TOEFL iBT are more academic in nature, both in terms of the content and the cognitive demands of the task. Some of the aspects of TOEFL iBT that make the test relatively more academic include extensive reading and listening passages and integrating information from multiple texts in speaking and writing. The report also highlights some of the ways in which a commitment to automated item generation and scoring limits the ability of a test like DET to fully represent the construct of academic language proficiency. Suggested citation: Cushing, S. T. (2025). Testing academic language proficiency: Comparing the TOEFL iBT® test with the Duolingo English Test (TOEFL Research Report No. RR-104). ETS.