Beyond the word count: A corpus-based analysis of vocabulary complexity, readability, and curriculum alignment in junior high school English tests

Abstract

Standardized English tests may impose linguistic requirements that go beyond the language competence they are designed to measure. This study analyzes the complexity of vocabulary, readability and suitability with the curriculum of English test materials for junior high school students in Tes Kemampuan Akademik Daerah (TKAD) in Special Region of Yogyakarta. It focuses specifically on (1) lexical frequency levels in the assessments, (2) the relationship between lexical complexity and readability, and (3) the alignment of vocabulary requirements with curriculum standards for Phase D students at around CEFR A2 level.  In this research, the method used was a quantitative descriptive method with a corpus-based approach and supported by exploratory correlational analysis. The research data were 20 TKAD English test packages with 93,242 tokens. Vocabulary was measured using AntWordProfiler and the New General Service List (NGSL). Lexical density and diversity were measured by TAALED and readability was measured by Flesch Reading Ease and Flesch–Kincaid Grade Level. Pearson correlations were computed at the assessment-package level (N = 20). The results showed a mean NGSL coverage of 64.93% and a mean off-list percentage of 35.07%. Lexical density, Measure of Textual Lexical Diversity (MTLD) and off-list vocabulary percentage were significantly correlated with both readability measures, with lexical density showing the strongest correlations. These findings indicate that vocabulary breadth and lexical complexity provide additional information to standard readability measures. However, the extent of NGSL coverage and formulaic readability metrics do not necessarily indicate the difficulties of learner comprehension or inconsistencies in curriculum alignment. The research emphasizes the significance of including lexical profiling, readability assessment and expert contextual review in quality control processes for assessments.