NSF/AAAS Conference Explores How to Measure Minority STEM Gains

In the race to raise the next generation of science and technology leaders, special programs and recruitment initiatives are flourishing around the country to attract minority scholars. But thoroughly assessing these strides and stepping-up the pace to meet urgent 21st century needs are still very much works in progress.

Even amid evidence of escalating numbers of Ph.D. degrees earned by minorities in science, technology, education and mathematics (STEM), advocates for such advancement are pushing academic institutions to collect more data on their efforts and vastly improve their evaluation techniques.

These expectations are at the heart of an ambitious collaboration between the National Science Foundation (NSF) and AAAS and were the focus of a conference run by the two organizations in Puerto Rico 26-28 January.

“The situation is: ‘What do we have to do within the institutions to maximize the possibility that those who enter with the talent to do this are actually there at the end?’” said Shirley Malcom, head of Education and Human Resources at AAAS and one of more than 150 science advocates, federal officials, scholars and administrators who attended the sessions. “You can say it is the students’ problem, but maybe it is the institutions’ problem.”

In many cases, colleges and universities are being asked to revamp the way they look at themselves and their graduate students, from recruitment to retention to the workforce. The Evaluation Capacity Building Meeting centered on practical ways of achieving these goals, and the critical importance of doing so.

“The way we engage and retain under-represented academic talent in the STEM fields will determine how well we can compete in the coming decades,” NSF Director Arden Bement said in a speech to conference attendees. [See the full text of the speech here.]

Bement said the effort to develop “comprehensive, sustainable measurements to identify, quantify and disseminate best practices” is critical to the mission.

“We need to make sure that the STEM fields are a mosaic that reflects all the cultures of this nation,” he said. “And make no mistake, our economic competitors will be right there to take our place if we stumble. Quite frankly, I believe that broadening participation among the ranks of the underrepresented is a matter of national security and economic survival.”

While colleges and universities have long collected a variety of information about their graduate students, it is often thin on race and ethnicity, spread throughout various departments and computer networks and not easily shared or compared among institutions. “It is decentralized and the data is not being collected thoroughly,” said Malcom. “People leave and you have no idea why.”

Over three days, conference participants compared their experiences and explored ways to build their evaluation capabilities — a central component of NSF’s Alliance for Graduate Education & the Professoriate (AGEP) program.

“What we are really interested in is improving the graduate education enterprise and recruiting more minorities into that enterprise,” said NSF’s Roosevelt Johnson, who directs the AGEP program. “The goal of AGEP is pretty simple: we need to increase the number of underrepresented minorities in STEM disciplines. The uniqueness of the program is in how we do it.”

Johnson said that “virtually every federal program has built into it some aspect of evaluation, but what you don’t get is a very coordinated comprehensive program-wide approach. And that is important because we are trying to have a national impact.”

AGEP focuses on increasing participation of African-American, Hispanic, Native American, Alaskan native and Pacific Island students Under the program, colleges and universities form working partnerships, called alliances, and receive five-year grants ranging from $500,000 to $1 million per year. The initiative had a $14 million budget last year and supports 22 alliances led by about 78 primary academic institutions around the nation.

AAAS has been involved since the effort began in 1998. Through NSF grants, AAAS is leading the program’s crucial data collection and progress evaluation component. The organization is helping grant recipients develop infrastructures for assessing their performance and adapting their efforts to changing trends and challenges. The AAAS endeavor is also meant to gauge the overall impact of the program and reveal lessons for adjusting and improving it.

“If you ask the question, ‘How many African American women advanced to candidacy,’ most institutions can’t answer that question,” said Yolanda George, deputy director for education and human resources at AAAS and the association’s point person for the AGEP program. “You can’t know what the progress is unless you have the specific data.”

The overarching goal of AGEP — boosting minority presence in the next generation of S&T professionals — has been readily grasped, but the experts involved in the program describe the data collection and evaluation as paradigm shifts for the institutions. “They have never been asked to get the type of data we are asking them to get,” George said.

During the conference, George and her colleagues released a framework document that spells out the assessment goals and helps guide the institutions through the process. The intent is to accumulate a wide range data over 10 years — specifically detailed information broken down along racial and ethnic lines and citizenship — and to encourage effective use of this data in everything from recruitment and publicity to student advising and department policies.

Institutions participating in AGEP are being asked to compile figures on who applies, who is admitted and who advances to PhD candidacy, on new enrollees and overall enrollees on who earned master’s degrees and who earned PhDs, and where the graduates end up in the workforce and professoriate.

“Special programs have gotten us to a point,” said George, “but if we really want to see some change then we really have to change the way colleges and universities do business in general.”

At the University of California-Irvine, the effort is well underway. The university already has an active minority science program that collects extensive data on recruitment, retention and progress toward the doctorate. R. Michael Mulligan, a professor and associate dean for Graduate Studies there, said that in the 2004-2005 academic year, the research showed a dip in underrepresented minorities in the School of Biological Sciences, from 12 to 7 students. The slide set off a recruitment drive and this year the number is back at 12.

“You never know if these are random changes or whether it is a trend and you are not doing something right,” said Mulligan, who shared his school’s experiences during one of the conference sessions. “So, we are trying to keep a database on how our students do and analyze the situation.”

Mulligan said the new data work can be extremely time-consuming for many AGEP alliance participants and that the hardest part is bound to be tracking students once they get their degrees. “It is an important objective for the agency to show how well the program works,” he said. “From my perspective, we do the best we can.”

Indeed, the program’s architects acknowledge the long-haul nature of what they are asking and insist that it’s more than a way to gauge the results of federal spending. Ultimately, they contend, thorough data collection and analysis will lead to a more robust STEM graduate enterprise overall and to a diverse and first-rate S&T workforce for the future.

“We are hoping that the institutions will recognize the utility of the efforts in a broader context then AGEP,” said Johnson, “and they will actually see it as a better way to do business.”

Andrew Metz

8 February 2006