Computer-enhanced science testing is becoming more popular as a way to assess knowledge and skills that ordinary paper and pencil tests cannot measure, but research on the actual effectiveness of online testing is limited.
Now, in a new study organized by AAAS and WestEd, a research and development agency located in California, researchers have confirmed the advantages of interactive, online science testing while also finding a number of areas where caution is warranted.
The study, funded by the National Science Foundation, and published in the April issue of The Journal of Research in Science Teaching, investigated three approaches to online science testing.
The study compared three versions of an online ecology test. | WestEd
The AAAS team was led by George DeBoer, deputy director of Project 2061, a long-term research and development initiative of AAAS focused on improving science education for all. Authors on the paper also came from the Smithsonian Institution and the Australian Council for Educational Research, as well as WestEd.
In theory, computer-based online tests can allow educators to more readily assess students' inquiry skills and understanding of complex material. In biology, for example, they can present animations of organisms interacting in an environment and let students manipulate data and design experiments to help answer questions about that environment.
But online testing also requires that students be familiar with the technology involved and adequately prepared for the interactive nature of the test.
"Online testing is becoming more common, but it is still not true that all students have had experience with online testing or that we can assume that all students will be equally impacted by a change to online testing," the researchers write.
As they note, students may not have much opportunity during typical classroom activities to develop the skills needed for a sophisticated online test, such as the ability to revisit observations, repeat experimental trials or confirm conclusions.
To counter the unfamiliarity students may have with taking tests in an online environment, groups that are moving toward online testing are also developing pre-assessment tutorials that students can use to prepare them for the new tests.
In the study, 1,556 middle-school students in 12 states took three different versions of an online ecology test on three consecutive days:
- A "static" test, comparable to a traditional paper and pencil test, that had still images and "yes" or "no" questions about the behavior of caribou, hares and other organisms in a tundra environment. Students were also asked to recognize correctly drawn food web diagrams and correctly designed experiments.
- An "active" test, with animations of animals interacting in a grassland environment. As part of the test, students were asked to draw food web diagrams themselves to help explain what they were seeing. They also got to run experimental trials and see the results of their investigations generated graphically.
- An "interactive" test, with simulations of a lake environment in which organisms compete for food and change in number over time. The students were offered line graphs and data tables to describe the population changes; they also were asked to create food web and energy flow diagrams. And the interactive test allowed them to design experiments and change data values to see what would happen as circumstances in the ecosystem changed.
Each of the computer-based tests was designed to assess the students' grasp of basic concepts in ecology, including that "producers" use energy from the sun to make sugars and that "consumers" eat other organisms or parts of organisms, along with related inquiry skills such as the students' ability to design and interpret experimental investigations.
Scores on the interactive version of the test were generally lower than the static, more traditional version. Students may have had difficulty with the more dynamic nature of the interactive test, the researchers write, and also may have been unfamiliar with some of the technology involved.
But the study did find that the interactive test enabled testing of more complex reasoning skills, a desired goal. And additional experience working in the online environment improved the students' performance on all three of the test types, especially the interactive version.
Given the additional cost of creating interactive online tests and the challenges that some students have with them, the authors say it is important that the complexity of the test be well-considered and appropriate to the knowledge being measured.
"As efforts to make use of online testing will inevitably continue," they conclude, "it is important for researchers in this area to continue to try to understand in more detail when it is worthwhile to develop interactive and active assessment items and when it is sufficient to rely on" the more traditional type of test.