One of the challenges of National Testing is the requirement to mark large volumes of test papers to a nationally agreed rubric in a relatively short period of time whilst maintaining a high level of consistency.

Part of the remit of the Australian Education Systems Officials Committee (AESOC), which comprises Chief Executive Officers for education and training from MCEETYA jurisdictions, is to constantly review methods of improving assessment and testing processes. In March 2008, they therefore approved a national trial of Automated Essay Scoring [AES].

The task of obtaining first-hand evidence of the reliability and validity of machine marking of students’ writing was passed to the Benchmarking and Educational Measurement Unit (BEMU) who conducted the project on behalf of the Performance Measurement and Reporting Taskforce (PMRT). For the purposes of this trial BEMU used IntelliMetric, an Automated Essay Scoring system developed by Vantage Learning which uses a combination of Artificial Intelligence, Natural Language Processing and statistics to perform the marking. IntelliMetric® has been widely used globally since 1998 to mark open-ended essay-type assessments.

The trial was conducted in WA, QLD and SA with approximately 3000 students from across years 5, 7 and 9 writing a response to a National Assessment Program – Literacy and Numeracy (NAPLAN) trial prompt. All essays were then marked by both expert markers and by IntelliMetric®.

As the data presented will show, the project clearly demonstrated that high levels of accuracy and reliability can be obtained from automated essay scoring and that it compares favourably with human marking. Possible benefits include improved reliability, cost savings due to reduced training and marking time, the ability to customise the rubric to suit institutional standards and receipt of instantaneous results.