VietTESOL International Convention 2026

Recent advances in large language models (LLMs) have increased interest in their use for automated writing assessment. Although previous studies have examined the extent to which AI-generated ratings align with human ratings, alignment alone may not adequately reflect effective rating behavior. In rater-mediated assessment, some degree of variation among human raters is expected due to individual differences, whereas LLMs may exhibit more deterministic scoring patterns shaped by their training and model architecture. This raises questions about whether LLMs apply rating criteria with the same flexibility and reliability as human raters. Therefore, this study examines the rating behavior of human raters and publicly accessible LLMs in the analytic assessment of L2 writing using a Many-Facet Rasch Measurement (MFRM) approach. The study uses 60 writing scripts produced by A2-B1 undergraduates at a university in central Vietnam, including 30 email responses and 30 narrative essays. The scripts are evaluated using a five-point analytic rubric assessing content, communicative achievement, organization, and language. Three experienced human raters and three LLMs (GPT, Claude, and Gemini) evaluate the scripts under standardized conditions using identical prompts, rubrics, and essay sets. Essays are evaluated in small batches (i.e., 5-10 scripts) using fresh chat sessions to reduce potential context and memory effects associated with publicly accessible LLM interfaces. Ratings are analyzed using MFRM to compare rater severity, fit, and criterion-level rubric application across human and AI raters. Findings are expected to contribute to discussions surrounding the potential and limitations of LLMs in L2 writing assessment and automated scoring practices.