AAAS has launched an innovative Web site with more than 600 multiple-choice test questions to help educators assess more precisely what students know about key ideas in science and—just as importantly—the incorrect ideas they have.
The new Web site, which also offers an unusually detailed picture of how middle and high school students across the United States are currently doing in science, features information on what they know and on hundreds of misconceptions they have about everything from the size of atoms to whether all organisms have DNA.
Knowing these misconceptions and how pervasive they are—which is not typically part of the analysis of test results from state testing or from leading national and international testing organizations—can help teachers improve instruction and better design their own test questions to assess whether students truly understand the science concepts they are being taught.
The newly developed test questions also counter the widely held view that multiple-choice questions are useful only for testing recall of memorized definitions and trivial facts, says George DeBoer, deputy director of Project 2061, the AAAS program that developed the new assessment Web site.
“As a result of our efforts, many of the test questions included in the new Web site measure not only knowledge of factual information, but they also probe a student’s ability to explain real-world phenomena, reason logically through problem situations, or identify the reason why a claim is true,” DeBoer said.
Multiple-choice testing has some clear advantages, he said. It tends to be less costly and easier to administer and score than other types of testing. “It is unlikely to disappear from the assessment landscape,” he said. “The goal of the project, then, was to improve on the design and use of multiple-choice questions.”
But the focus on multiple-choice testing should not be misinterpreted as an endorsement of it as the best approach to student assessment, DeBoer cautioned. “Any type of assessment can be used effectively,” he added, “as long as attention is paid to certain fundamental design principles.”
The new AAAS Web site presents detailed information on how a national sample of students answered each question, along with an analysis of both their correct and incorrect responses. In contrast, national and international assessment programs such as the National Assessment of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and Program in International Student Assessment (PISA) report on the percent of students in participating states or countries who answer items correctly but do not report on the incorrect ideas students have, DeBoer said.
Project 2061, founded in 1985 by AAAS to improve science education, developed the assessment items and collected data on them under a grant from the National Science Foundation. The Web site, where the results of this work can be found, includes data on student performance on the test questions by gender, grade level, and whether or not English is the student’s primary language.
There was “essentially zero” difference in how well girls and boys answered the questions, said DeBoer, with only two-tenths of a percentage point separating their scores.
Students whose primary language was not English scored seven percentage points lower on the test; DeBoer said his research team has not yet analyzed the testing data by student race or ethnicity.
Each question typically was answered by at least 2000 students in field tests involving school districts across the nation. In 2010, for example, more than 90,000 students in 814 schools participated in the field tests. Project 2061 researchers also conducted on-site interviews with students to gauge the effectiveness of the questions.
Constructing good multiple-choice questions requires knowledge of science and of effective assessment practices. “Too often,” DeBoer said, “test questions are not linked explicitly to the ideas and skills that the students are expected to learn.” And too often the questions are written in a way that makes it difficult for students to understand what is being asked, he said.
With increasing calls for national science standards and a common core curriculum for middle school and high school students, it is more important than ever to get assessment right, DeBoer said. “Good assessments can be used to actually improve students’ learning and not just to hold teachers and schools accountable,” he added. “Assessments that are designed to diagnose students’ misconceptions can be powerful educational tools.”
For the key idea that “all matter is made up of atoms,” the AAAS Web site notes that 27% of the middle school students and 20% of the high school students who were tested incorrectly believed that “cells are not made up of atoms.” On the key idea that “genetic information is encoded in DNA molecules,” 40% of middle school students and 30% of high school students had the misconception that only animals have DNA while plants and mushrooms do not.
Having information about such misconceptions is just as important as knowing how many students answered a question correctly, DeBoer said. It can reveal gaps that prevent a coherent understanding of the topic.
The AAAS Web site is organized to quickly give users a picture of what students know and the misconceptions they have. The test questions for a specific science concept are listed together, arranged from the highest to lowest percent correct. The misconceptions also are listed in order, with the misconceptions selected most frequently at the top of the list.
So, for example, there are 20 questions probing the idea that “genetic information is encoded in DNA molecules.” At the top of the correct responses, 94% of middle school students knew that “DNA is the molecule that contains genetic information that is passed from parents to offspring.” At the bottom of the list, only 15% of the students knew that “Genes are chemical sequences of nucleotides, not the actual expression of characteristics or traits.”
“This is extremely valuable information for teachers and curriculum developers to have,” said DeBoer, “because it shows them where instruction needs to be targeted.”
The Web site has received early positive reviews from teachers and curriculum specialists who attended a recent workshop at AAAS on the methods used to develop the test items.
The emphasis on student misconceptions is particularly helpful, said Deagan Andrews, a curriculum and assessment specialist in the Greeley, Colorado, school district. Referring to some of the major standardized test programs, he said: “No one releases any information about misconceptions. They are interested in whether students got it right or wrong.” He said the emphasis on misconceptions “is a critical piece that has been missing,” offering teachers an opportunity to “think about their instruction and what they may be doing that may be perpetuating misconceptions.”
“Students create strange conceptions about the world from their experiences,” agreed Anu Malipatil, a school administrator for a network of charter schools in New York and Connecticut. “It becomes more difficult to teach students without actually addressing the misconception first.” She said most state tests do not provide an analysis of incorrect answers. The AAAS test items, she said, can be used by teachers to diagnose potential problem areas and find out how many students in a classroom have misconceptions about key concepts.
Teresa Eastburn, a former classroom science teacher who is now an education and outreach specialist at the National Center for Atmospheric Research in Boulder, Colorado, said the assessment site ultimately benefits teachers as well as students, particularly teachers who may not have an extensive background in science. Because of the level of detail, including the emphasis on misconceptions, the site “will really inform teaching,” Eastburn said.
Some Things Middle and High School Students Know and Misconceptions They Hold
Some things students know (number in parentheses is % who answered correctly)
- DNA is the molecule that contains the genetic information that is passed from parents to offspring. (95%)
- Changes in a population of organisms in a food web (worms) can affect the population of its predator (robins). (84%)
- Animals compete for resources when they are limited. (80%)
- The lungs both take in oxygen molecules and eliminate carbon dioxide molecules. (75%)
- Populations increase when the number of births is greater than the number of deaths. (73%)
- Children need food as a source of energy and as a source of material for building or repairing body structures such as muscles. (66%)
- Both wind and rain can wear away the solid rock layer of a cliff. (65%)
- Matter is made up of atoms. (64%)
- A car has the most motion energy when it is traveling at the highest speed. (64%)
- Both ocean basins and continents are part of earth’s plates. (64%)
- A blood cell is bigger than an atom. (63%)
- Increasing an object’s temperature increases its thermal energy. (62%)
Some frequently held misconceptions (number in parentheses is frequency of selection of this misconception across items)
- The different cell types (skin, muscle, cartilage, etc.) found in a given individual’s body contain different DNA. (58%)
- Each cell contains only the specific genetic information required for its function. (43%)
- The heart is the mixing place for air and blood. (40%)
- Molecules from food are distributed by way of special tubes to the rest of the body, not by way of the circulatory system. (38%)
- Air is distributed through the body in air tubes. (43%).
- Under the influence of a constant force, objects move with constant speed. (44%)
- The atoms of the reactants of a chemical reaction are transformed into other atoms. (40%)
- New atoms are created during chemical reactions. (40%)
- Atoms can be destroyed during a chemical reaction. (36%)
- When a liquid in a closed container is heated, the mass of the liquid increases as the liquid expands. (41%)
- During biological decomposition in a closed system, the total mass of the system decreases. (37%)
- Earth’s plates are located deep within the earth and are not exposed at the earth’s surface. (49%)
- Mountains form by the piling up of pieces of rock. (52%)
From the AAAS Science Assessment Web site
Visit the AAAS Project 2061 Science Assessment Web site.
Learn more about Project 2061, AAAS’s science education reform and science literacy initiative.
Listen to a 7 April teleconference call with reporters regarding the new AAAS Assessment Web site. Or download the MP3.
Read lists of some things middle and high school students know about science and some of their frequently held misconceptions.