Speaker
Description
Academic writing assessment is crucial for students who want to know their proficiency level and improve based on the feedback. Due to the high demand, the number of learners requesting analytic scores and direct feedback from teachers has increased, putting pressure on teachers’ workload. In response, many researchers have applied LLMs to this process, which reduces the burden of English teachers. However, the concerns of AI tools and the quality of feedback still remain. This pilot study was to investigate the reliability of ChatGPT, Gemini, with benchmark expert scores from five published IELTS sample essays from Cambridge IELTS materials, measuring intra-rater and inter-rater values. Moreover, this pilot study also measures the rubric coverage of AI-generated feedback. The intra-class correlation coefficients (ICCs) showed that intra-rater values of Gemini (ICC=0.94) were slightly higher than ChatGPT (ICC=0.91) across all criteria and overall scores of existing essays. Likewise, the overall inter-rater values of Gemini (ICC=0.93; 80% within ±0.5 band) were also higher than ChatGPT (ICC=0.88; 60% within ±0.5 band), despite both AI-generated scores tending to be lower than the human score. The rubric coverage ratios showed that, in general, most AI-generated feedback was vague. The errors were mentioned, but not suggested how to fix. Furthermore, the AI-generated feedback tended to recommend the direct correction at the surface level only (vocabulary and grammar), whereas it often provided indirect and metalinguistic types. Thus, we observed that LLMs show promise as a low-stakes support tool for writing assessment, which is more suitable for use in self-practice and a classroom-based system.
Keywords: writing assessment, LLMs, reliability, rubric coverage, IELTS writing
Biography
I am currently a master student at Khon Kaen University Thailand with the major is Applied Linguistics. In the light of the proliferation of technology, advancements in generative AI, such as large language models (LLMs) - a new generation of AES, may serve as a potential solution to the burdensome task of essay evaluation often faced by teachers and students. But some some concerns about the reliability of AI-generated scores still remain, as well as the feedback from AI tools. I want to conduct this pilot study to measure these values as a empirical evidence for the development of AI in ESL field.
| Affiliate type | University |
|---|