Best Statistical Practices to Promote Research Integrity

By John S. Gardenier

Dr. John S. Gardenier is a researcher, ethicist, and guest lecturer in Research Ethics. He can be reached at drgarden@erols.com.

  “The statistician’s role in a biomedical research project is to drive the p-value below 0.05"
Generally accepted practice?  Research misconduct?  Both?

 The Concept of Research Misconduct

 Over the 20^th century, there developed an expanding notion of things that are unacceptable in science.  Deliberate hoaxes were seen as moral failures early in the century, but differential treatment of human subjects of research based on race, ethnicity, or intelligence were often accepted.  The Nuremberg Trials and the Helsinki Universal Declaration on Human Rights contributed to a sea change.[i]  That change continues in the evolution of Institutional Review Board (IRB) practices.  Separately, a concept of definable “research misconduct” developed late in the century.[ii]  It was, arguably, limited to *fabrication or *falsification of data plus *plagiarism of exact wording without permission or attribution. These asterisked items are collectively known as FF&P.  Some dissenters argued that mere FF&P leaves ample room for malfeasance.  In 2000, a common federal definition emerged to include broadened concepts of FF&P plus whistleblower protection.[iii]

 “It Will Not Work.”

At a Town Meeting on the then-proposed definition of research misconduct held at The National Academies on November 17, 1999, all scheduled speakers praised the definition as a virtually optimal solution to any remaining problems of research integrity.  Some scientists in the audience, especially those from biomedical research disciplines, found it excessively harsh with regard to inclusion of matters of omission, protection of whistleblowers and standards of proof.[iv] Others felt it was insufficient.  I noted that it did not clearly prohibit deliberate misuse of statistical methods to present false results.  When I quoted the aphorism on p-values (above), the reaction of that auditorium full of distinguished scientists was to laugh, despite the clear contempt for statistical honesty.  This is dangerous because misuse of statistical methods can be harder to detect than falsification of data and thus be more tempting and less risky to the perpetrator.[v]  If caught, one can always claim “honest error” due to a mistaken understanding of the statistical methods.  (Institute of Medicine (IOM) member, John C. Bailar III, has long deplored the varying effects of bad statistics on biomedical research; personal discussions with him over several years have greatly influenced my own thinking.)

C. K. Gunsalus, a member of the Ryan Commission that recommended a greatly revised concept of research misconduct[vi] -- and an experienced official of the University of Illinois -- explained why she felt the research misconduct definition “will not work.”  In her experience, institutional colleagues are generally reluctant to brand any actions as “research misconduct” even when they are recognized as immoral, disgusting and scientifically wrong.  Absent a finding of “research misconduct,” any deficient scientific practice such as incompetence, sloppiness, misleading reporting or abuse of colleagues may be excused as “honest error” or acceptable academic conduct. We need a more robust concept of responsible conduct in scientific research than the dichotomy between a finding of research misconduct and a presumption of total innocence.

Such a concept is entirely compatible with the IOM’s 2002 report, Integrity in Scientific Research: Creating an Environment That Promotes Responsible Conduct.  Among other conclusions, that report finds that “No established measures for assessing integrity in  the research environment exist.”[vii]  The proposal herein seeks to address those aspects of an environment that seek to avoid misuses of statistics and to promote best statistical practices.

A Taxonomy of Statistical Practices in Science

As noted by Sigma Xi in 1997,[viii] “No scientist can avoid the use of such [statistical] techniques, and all scientists have an obligation to be aware of the limitations of the techniques they use, just as they are expected to know how to protect samples from contamination or to recognize inadequacies in their equipment.”

This taxonomy is anchored at the high end by the concept of “best statistical practices” and at the low end by the common federal definition of “research misconduct.”  Best statistical practices are easy to define, at least in concept.  They involve professional expertise in statistical theory, methods and ethics as well as in the subject matter specialties of the research at hand.  (Not all such expertise need reside in one person, of course.)  In addition, researchers or research teams must apply themselves with diligence to assure: a thorough understanding of the research issue(s) and prior results, the relevant data, relevant sampling principles, the most applicable statistical methods, rigorous design and conduct of the data collection and analysis plus careful interpretation of results, and full and open reporting to support peer review and replication.  Finally, the researchers should demonstrate adherence to ethical principles and applicable best practice definitions, both in statistics and in the subject matter discipline(s).[ix]

The full taxonomy can be outlined as follows:

 Recommendations for Improving Research Integrity in Statistical Practice

 In keeping with the thrust of the IOM report on research integrity and in employing the taxonomy above, here are recommendations for steps to be taken by research institutions, funding sources, journals, and statistical software makers.  These would contribute to improving the overall climate of research integrity.

 As for the Rest of Us

Ultimately, we as scientists can never rely on the “authorities” to make things right, can we?  We all share a responsibility for critical thinking and critical appraisal.  We all have to understand at least the elements of statistical theory and statistical thinking and apply them to what we read as well as to what we do.  Recognize that bad statistical work is seldom truly attributable to “honest error,” and it can sink to the level of actionable research misconduct.  Do not accept inferior statistical work from colleagues, publications, or ourselves.  Never be afraid or embarrassed to ask for statistical help when needed.

Above all, we must recognize that sound science can never result from unsound statistics. 

References

IN MEMORIAM 

(Photo Courtesy of Edge Foundation, Inc. - http://www.edge.org)

Dolly “D” sheep, the first mammal cloned from an adult mammary somatic cell by nuclear transplantation and whose birth heralded a breakthrough in molecular biology that incited consternation over the morality of cloning life, died February 16 at the Roslin Institute in Edinbugh, Scotland after suffering from progressive lung disease.  She was 6. 

Since her birth on 5 July 1996, Dolly ewe-d and awed scientists and the public the world over with her sui generis pedigree: a genetic facsimile of her Fin Dorset sheep “mother.”  Like the country music star she was named after—Dolly Parton—one could say that Dolly (“D” sheep) was destined for stardom.   

Although her birth was unconventional, Dolly appeared not so different from other little lambs.  On the contrary, she lived anything but a quiet life, reaching celebratory status as evinced by international headlines.  And like most Hollywood celebrities she grew up in front of the camera, always on display for an admiring public.  Her august presence was the focus of the limelight, though almost eclipsed by the birth of her first lamb, which convinced doubters that cloned mammals can reproduce vibrant offspring the natural way. 

 Her life, however, was not all glamour and glitz.  While narrowly escaping the devastating foot-and-mouth disease that swept through the United Kingdom in 2001 by quarantine, Dolly’s health was failing. She was diagnosed with arthritis and later progressive lung disease that ultimately led to her termination.  Was the proof in the pudding—that her unique genetic dossier was to blame for the pangs of premature aging? No one knows just yet, as the postmortem investigation is ongoing.  Nonetheless, the world anxiously awaits the results.   

For better or for worse, Dolly will be remembered by history’s pages as the archetype of a scientific paradigm shift.  As such, she has left a mark, or more precisely a hoof-print on the sands of time, as an embodied artifact dressed in sheep’s clothing.  For Dolly “D” sheep will not simply be relegated to remembrance, a relic of the past.  Rather, she will be visible to all, as she is to be stuffed for display in the national museum in Edinburgh, perhaps as a monument immemorial to the public interest: as Scotland’s very own “Dollywood.” 

 Quiescat in pace agnus Dei. 

 Dolly is survived by her six lambs and her estranged suitor, David, the Welsh mountain ram.  *KA

On January 15, 2003 the Association of American Universities (AAU) released a statement urging academic institutions not to lose sight of their commitment to academic freedom as America prepares itself for war.

This statement entitled, “The Responsibility of Universities at a Time of International Tension and Domestic Protest,” voiced the concern that, “Supporters of various causes and points of view seek adherents and vehicles for their messages, including our universities themselves.” It stressed the importance of university protection of free speech and the maintenance of open environments for learning during times of war.  Also, the statement urged universities to further their responsibilities in two additional areas in order to prepare their campuses for possible adverse events.

First, AAU suggested universities should actively promote informed dialogue, analytical thought and exemplary arguments via appropriately guided information sessions. “It is incumbent upon the university to offer the expertise and experience of its faculty and staff members to broad audiences on campus through fora such as teach-ins and seminars,” the statement said.

Second, AAU would like to see universities establish clear boundaries for public protest, being particularly careful to outline the consequences for illicit disruptions such as violence, prevention of class, takeover of buildings, and email fraud, all of which should be prohibited.

The statement asked university presidents, faculty, and students to reaffirm their underlying reasons for being on campus: the work of learning, teaching, scholarship and research, uninterrupted by anyone.

“For the university to fulfill its obligations to academic freedom and to intellectual development, it must provide a forum in which individuals and groups can advocate their views,” the statement stated. “It must assure an environment for civil discourse to take place free from violence and intimidation.”

The statement by AAU can be found at: http://www.aau.edu/resources/aaustate1.15.03.html. *MP

In late February of this year, the National Institutes of Health published its final statement on sharing research data.  The statement is a result of a NIH review of the comments regarding the draft statement, which began in March of 2002.

The final statement differs from the draft statement in three instances.  The first change pertains to the effective start date of the policy changed from January 1, 2003 to October 1, 2003.

Second, applicants seeking $500,000 or more in direct costs in any year of the project period are expected to contact NIH program staff prior to submission and to include a data sharing plan in their application.  The plan should state how they will share their data, and if data sharing isn’t possible, an explanation of why that is so.

Third, in response to objections by several groups and individuals to the sharing of research data prior to publication, NIH has modified the definition of “the timely release and sharing” to be no later than the acceptance for publication of the main findings from the final data set.  This change was made to recognize the effort of the investigators who collect the data, but NIH insists that investigators may not benefit from prolonged exclusive use of data.

Public sharing of research data is a long-standing policy of NIH, which believes that the practice is essential for advancing research results and improving human health. However, NIH recognizes that the rights and privacy of research participants in NIH-sponsored research must be protected at all times and that data be free of identifiers before it is shared.

For additional information on NIH data sharing, see http://grants.nih.gov/grants/policy/data_sharing/ . *BP

On February 10, 2003, the RAC met and discussed the findings that a second child involved in the X-SCID study had developed leukemia. Dr. Alain Fisher, who began this study nearly three years ago, joined the conversation briefly via a teleconference voicing his concern for those children still remaining in the study and for the future of gene transfer therapy in general. At the end of the day, the RAC concluded that the gene transfer was a cause of both leukemias, noting that there could be other causes as well. The RAC suggested, pending further data or extenuating circumstances that retroviral gene transfer for X-linked SCID be limited to patients who have failed identical haploidentical stem cell transplantation or for whom no suitable donor can be identified. They found that resumption of non-X-linked SCID gene transfer studies may be justified contingent upon appropriate informed consent and monitoring plans. Finally, they concluded that there is not sufficient evidence at this time to warrant cessation of other retroviral gene transfer studies, which would be contingent upon appropriate informed consent and monitoring plans as well.

More information about the RAC and about the X-SCID gene transfer studies can be found at http://www4.od.nih.gov/oba/rac/meeting.html *MP

Despite its widespread use and costs, the Cochrane Collaboration has found that “there is little empirical evidence to support the use of editorial peer review as a mechanism to ensure quality of biomedical research.” Editorial peer-review is used world-wide as an quality assessment instrument in the selection of submissions to biomedical journals.

The Cochrane report titled “Editorial Peer Review for Improving the Quality of Reports of Biomedical Studies” reached this conclusion after reviewing the effects of the processes of editorial peer-review on original research studies submitted for paper or electronic publication in medical journals.

Results show that factors such as a referee’s training, electronic communication media, and the practice of concealing the identities of peer-reviewers and authors have little or no effect on the outcome of the quality assessment process.

Similarly, in an attempt to address current questions pertaining to the benefits of the editorial peer-review process and public concern regarding reliability of scientific information in the media, the Royal Society, Britain’s leading academic scientific institution, will investigate the peer review process of scientific literature.  It will consider possible failings in the current mechanism for peer review, explore alternatives, and provide direction for the general public on judging the significance of a report on new research.

The Society will also address concerns relating to research results leaking to the media before undergoing peer review. The culmination of the investigation will result in two chief documents – a set of guidelines outlining proper methods of releasing results of scientific research and a “Science Brief” offering practical advice to the public on interpreting the importance of scientific results.

An abstract of the Cochrane report is posted at http://www.cochrane.org

The Royal Society can be found at http://www.royalsoc.ac.uk *BP

TOWARDS A BROADER VIEW OF ACADEMIC ETHICS

by Brent Garland

The privileged position that academics hold in modern American society carries with it corresponding professional responsibilities, and Neil Hamilton has done the academic community a significant service with his new book on academic ethics. While many academicians receive substantive training in the ethics of their discipline (science, engineering, law, etc.), too often the ethical issues of life in the academy are all but ignored. Many young faculty receive a day or two (or if lucky, a week) of training at the beginning of their careers that is related to appropriate interactions with students, administrators, and colleagues. All too often, the "ethical lessons" are boiled down to three points: don't plagiarize, don't have sex with your students, and don't steal from your grant funds. While these points are undoubtedly good advice, the failure to provide substantive education on the ethical practices and principles of academia as a whole does students and faculty a grave disservice. Graduate students, new professors and experienced faculty alike can benefit from Hamilton's thought-provoking analysis of the ethical duties and correlative rights of academics, both as individuals and as a collegial body.

Tightly organized in a format that makes it ideal for a graduate seminar or an ongoing faculty discussion group, Hamilton presents first an analysis of the duties and rights of faculty, supplemented and illuminated by comparisons among some relevant documents, including the 1915 General Declaration of Principles of the American Association of University Professors (AAUP), the 1940 AAUP Statement of Principles on Academic Freedom and Tenure, and the 1998 Statement on Institutional Governance by the Association of Governing Boards (AGB). Hamilton uses these documents to help provide a historical and cultural context for the development of current concepts of academic ethics.

In his analysis, Hamilton lays out a framework for considering the ethical behavior of academicians, focusing on the right of the faculty to free speech, peer review, and shared governance, and the duties of professional competence and ethical conduct. Hamilton follows this analysis with chapters that borrow the "case method" of business and legal education. Each chapter contains numerous vignettes (which Hamilton terms "problems") that pose ethical questions for those involved, followed by series of discussion questions. The chapters are organized by the types of problems they present in Hamilton's framework: the duties of individual professors; the rights of individual professors; the duties of professors as a collegial body; the rights of the collegial body; and the rights of students.

Hamilton's choice to use a "case method" format should be effective for maximizing discussion, and is a real asset in helping to provide numerous examples of potentially problematic situations. The questions that follow each "problem" serve to identify and frame key ethical issues present in each vignette.

Following the "case method" chapters are a rich and useful set of appendices that contain the AAUP and AGB documents relevant to the initial part of the book. Provision of these texts are a useful tool for the reader who wishes to consider Hamilton's analysis in a more full context, or for faculties seeking to draft their own statements on similar academic ethics issues.

If this book has one weakness, it is that Hamilton's chapters on "problems" consist solely of that: problems, followed by questions, with no additional analysis or commentary by the author. Given Hamilton's expertise and deep thinking evinced on the subject in the initial chapters, readers could likely benefit more from his knowledge if he offered some model analyses of a subset of the problems presented. That said, Hamilton's book is a valuable resource to anyone with either a scholarly or practical interest in academic ethics.

Hamilton, Neil W. Academic Ethics. Problems and Materials on Professional Conduct and Shared Governance. American Council on Education/Praeger Series on Higher Education, 2002.

The first scenario asks Who Is a Parent? Reproductive technologies intended to help infertile couples have a baby are now a familiar and widely accepted practice. Nevertheless, the various permutations of donor sperm, donor egg, and donor uterus pose continuing challenges for family law, challenges that are playing out in courtrooms and in the lives of adults and babies. The cases presented in this segment address the following dilemmas: 

·        The baby with five parents: A married couple with fertility problems is unable to have a baby of “their” own following six attempted in vitro fertilizations and six surrogates. They eventually “conceived” a baby with donor sperm, donor egg, and a surrogate mother. However, during the pregnancy the husband decides to divorce his wife, renouncing all parentage and responsibility for the yet-to-be-born child. Is he the legal father with concomitant responsibilities?

·        The babies with two mothers: Wishing to have a family, a lesbian couple “conceives” three children using eggs from one partner and the uterus of the other, plus donor sperm. Do the two “mothers” have equal legal rights and responsibilities in relation to “their” children?

·         The twins whose “parents” change their minds: An infertile couple hires a surrogate to bear a child for them, a woman who was a surrogate for them on a previous occasion and who has also been a surrogate for other couples. Pregnant with twins, the surrogate learns that the couple is having marital difficulties and intends to place the twins in foster care. The surrogate wants to find a permanent, adoptive home for the twins. Does she have any rights regarding what happens to “her” babies?

The second scenario asks What Is Human? This scenario presents two provocative cases that illustrate the ethical and legal difficulties that can arise with technologies that allow the combining and crossing of species. 

·        A developmental biologist has filed a patent application on an embryo that is a chimera, part human and part chimpanzee. Increasingly uneasy about the directions in which research in his discipline are heading, he has taken this action to force a public review of the patentability of living organisms. Stated simply, how many cells does it take to make an organism human?

·        A man desperately ill with Parkinson’s disease agrees to a procedure that implants pig cells into his brain. Although his medical condition is dramatically improved, what does it mean to have the cells of another species functioning in his body?

The final scenario asks Who Has Rights? While genetic testing holds great promise for improving our lives, our society has yet to reach a consensus about when such testing is useful, even desirable, and when it is harmful or poses an intrusion into an individual’s privacy. Two railroad workers, along with more than a hundred of their co-workers, file for workman’s compensation due to carpal tunnel syndrome that they claim is a result of the repetitive motions required by their jobs. Their employer takes blood samples and, unbeknownst to the employees, conducts genetic testing on the blood. Who has a right to our genetic information and how can they use this information?

Bloodlines offers no easy answers to these questions, but it does provide an opportunity for viewers to ponder them. This series of ethical, social, and legal dilemmas will best be used in classrooms and other educational settings.

For more information see www.backbonemedia.org/bloodlines.