The Australian Curriculum Assessment and Reporting Authority (ACARA) has revealed key findings from its research, funded by the Federal Government, that online NAPLAN testing will deliver better quality results that will benefit schools, parents, teachers and students.
The National Assessment and Surveys Online Program, Tailored Test Design 2013 Study also found that as many as 50% of students will experience an enhanced assessment experience, particularly for high- and low-achieving students.
“The tailored test design and online assessment in general has been proven effective in the NAPLAN context,” Dr Stanley Rabinowitz, ACARA’s general manager – assessment and reporting, said upon the research’s release.
“We will get better quality results that will be returned quicker which will benefit government, schools, parents, teachers and students. We expect to see more engaged students and better informed teachers and parents.”
According to ACARA, as students concluded the trials of online tailored test design, they reported feeling more positive and accomplished, regardless of their abilities and educational background. The trials also provided an opportunity for students to be assessed by tests catering to their needs, as well as more accurate and timely diagnostic information about student learning needs.
“ACARA is continuing research into online assessment and further research results will be available in 2015,” Dr Rabinowitz said. “The research allows ACARA, with our partners, to continue to fine tune our approach to moving assessment online. We need to keep investing in the research so that when the Education Council makes its decision about the country’s move to online assessment, we are ready to go.
“The sooner we get online, the quicker the benefits can appear for governments, schools, parents, teachers and students. This research shows us the future of assessment for Australian school students and we know it works. We now need to get on and implement it.”
Federal Education Minister Christopher Pyne said the Government is committed to moving NAPLAN online.
“ACARA’s research suggests that by encompassing a ‘tailored test design’ – the assessment could deliver questions based on a student’s answer to the previous question,” Pyne said in a statement. “This would provide students better opportunities to demonstrate their knowledge, as well as increase their engagement with tests and their sense of achievement.
“Moving NAPLAN online will improve the quality of student performance data at the national, state and territory level.
“Online delivery of NAPLAN will also provide significantly faster turnaround on student results. Schools and parents will be able to see results sooner, and develop appropriate strategies to support and extend their students’ literacy and numeracy skills.”
ACARA: Background to the research program
In 2012 ACARA was funded by the Australian Government Department of Education to conduct research to inform decisions on the transition of the National Assessment Program – Literacy and Numeracy (NAPLAN) from paper-based to computer (online) assessments.
The initial phase of research, undertaken in 2012, investigated how the test delivery mode (paper vs. computer) affected student performance in and engagement with the current NAPLAN tests. The second phase of research, conducted in 2013 and summarised here, investigated the feasibility of the enhanced adaptive test design for NAPLAN online assessments.
ACARA: Multistage adaptive test design (the ‘tailored test design’)
Multistage adaptive test design (the ‘tailored test design’) is where the test difficulty is adjusted to students’ needs. After students respond to a set of test items, they are provided with a subsequent set of items that best suit their observed ability (i.e. achievement level). In the proposed tailored test design, students go through three stages containing item sets (testlets) of varying difficulties to complete the test, as illustrated in Figure 1.
ACARA: 2013 Main Tailored Test Design Study
During 2013, more than 250 schools of satisfactory diversity participated voluntarily in the main tailored test design study. Over 2500 students in Years 3 and 5 and 1500 students in Years 7 and 9 completed either numeracy or reading online tests. Most of the testlets used in this study were created from existing NAPLAN test items that had been rendered to suit the online delivery mode.
ACARA: Related studies
In a related study, students from 16 mainstream schools participated in structured interviews to ascertain how students reacted to the rising and falling pattern of item difficulty, a crucial feature of the tailored test design. The same methodology was used in a separate study that investigated whether the tailored test design can accommodate the assessment needs of students with socio-educational disadvantage in numeracy tests.
In another study, ACARA collaborated with the Northern Territory Department of Education to collect information about the extent to which the proposed tailored test design provides a better testing experience for Indigenous students and students in remote communities. In this study, online tests were administered in eight Northern Territory schools, including two very remote schools.
The key findings from these studies are as follows:
- Results of the tailored test design studies show that the delivery of multistage branching tests for NAPLAN online is sound and feasible, and that these tests offer better measurements of student performance, particularly for high- and low-achieving students.
- The results show that the current measurement model can be used to construct a NAPLAN online measurement scale.
- The psychometric analyses also show that further work is required to finalise the measurement aspects of the tailored test design; in particular, testlet boundaries require further refinement.
- The tailored test design and the proposed branching mechanism work effectively to adapt to the different ability groups. Consequently, well-targeted tests can be administered to different ability groups, thus increasing measurement precision.
- The investigation of cognitive and behavioural engagement of students with the tailored test design showed that multistage testing will provide an opportunity for all students to be assessed by tests catering more fully for their assessment and learning needs.
ACARA: Third phase of the research program
In August 2014 ACARA began the third phase of its research program with the aim to further refine the measurement aspects of the tailored test design for reading and numeracy tests. This study also includes trialling of grammar, punctuation and spelling test items delivered online, including students listening to spelling items through headphones. The results of this program will be made available in 2015.