Case Study Module

Scientific Rigor and Reproducibility (SRR)
Case Study Module

Preface

The concept of reproducibility – that the same experiment always gets the same results no matter who performs it – is a cornerstone of scientific objectivity.¹ In an era of increasing pace and complexity in biomedical science, however, there is concern now highlighted in several high-profile reports that reproducibility is less than that which inspires unequivocal trust from scientists and the public. While it has long be asserted that mistakes are culled as scientists move forward, i.e. that science is self-correcting, confidence in the argument is considerably less than it once was.

Problems in reproducibility are viewed to stem from problems inherent to elements of experimental design and communication of that design to the scientific community. Some contend that high on the list of factors contributing to these problems are deficits in training. Others argue more broadly. But it is important to note that problems in reproducibility are not to be confused with those that might arise through misconduct in research. It is probably for this reason, that NIH is reluctant to include reproducibility as a formal RCR topic.

NIH has issued a notice of intent to clarify and revise instructions for research grant applications, and to do the same for review criteria, to enhance reproducibility of research findings through increased scientific rigor and transparency. It has also issued a notice of a coming requirement for formal instruction in rigorous experimental design and transparency to enhance reproducibility. These notices focus on i) scientific premise, ii) rigorous experimental design, iii) consideration of sex and other relevant biological variables, and iv) authentication of key biological and/or chemical resources.

¹Trouble at the lab. The Economist. October 19, 2013. http://www.economist.com/node/21588057/

Background

A short article by Francis Collins and Lawrence Tabak (Nature 505: 612-613, 2014) provides a good summary of problems likely underlying irreproducibility and NIH thoughts on how to surmount these problems. The link to this article is the following: http://www.nature.com/news/policy-nih-plans-to-enhance-reproducibility-1.14586

Policies and Guidelines

The University has no specific procedures or guidelines regarding reproducibility apart from those related to scientific misconduct. NIH provides guidance through a variety of notices – those of relevance are [NOT-OD-15-103] (Enhancing reproducibility through rigor and transparency) and [NOT-OD-16-034] (Advanced notice of coming requirements for formal instruction in rigorous experimental design and transparency to enhance reproducibility).

Provided here for the sake of definitions are excerpts (abridged) of a resource chart provided by the NIH relating to rigor and reproducibility in NIH applications (http://grants.nih.gov/reproducibility/index.htm):

Scientific Premise. The scientific premise as it relates to a grant application is the research that is used to form the basis for the proposed research question(s). The applicant should be able to describe the general strengths and weaknesses of prior research being cited as crucial to support the application. Items for discussion may well include the rigor of previous experimental designs, as well as the incorporation of relevant biological variables and authentication of key resources.

Scientific Rigor (Design). Scientific rigor is the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, and interpretation and reporting of results. In a grant application, one would emphasize how the experimental design and and methods proposed will achieve robust and unbiased results.

Biological Variables. Biological variables such as sex, age, weight, and underlying health conditions, are often critical factors affecting health or disease. In particular, sex is a biological variable that is frequently ignored in animal study designs and analyses, leading to an incomplete understanding of potential sex-based differences in basic biological function, disease processes and treatment response. In a grant application, one would explain how relevant biological variables, such as the ones noted above, are factored into research designs, analyses, and reporting in vertebrate animal and human studies.

Authentication. Key biological and/or chemical resources include, but are not limited to, cell lines, specialty chemicals, antibodies, and other biologics. In a grant application, one would describe the methods to ensure the identity and validity of key biological and/or chemical resources used in the proposed studies.

Case Studies

Access to case studies requires a PennKey. They are available here.

Comments/Resources

Resources for rigorous experimental design and transparency to enhance reproducibility (‘REDTER’) continuously evolving. We’ll do our best to keep the list up to date yet manageable.

Here are some papers:

Arrowsmith, CH, Audia, JE, Austin, C, Baell, J, Bennett, JB, Bountra, C, et al. The promise and peril of chemical probes. Nat. Chem Biol. 11, 536541 (21 July 2015)
Lorsch, JR, Collins, FS, Lippincott-Schwartz, J. Fixing problems with cell lines. Science. 346, 1452-1453. (19 December 2014).
Clayton, JA, Collins, FS. Policy: NIH to balance sex in cell and animal studies. Nature. 509, 282-283. (15 May 2014).
Collins, FS, Tabak, LA. Policy: NIH plans to enhance reproducibility. Nature. 505, 612–613. (30 January 2014)
Landis, SC, Amara, SG, Asadullah, K, Austin, CP, Blumenstein, R, Bradley, EW, Crystal RG, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 490, 187-191. (11 October 2012).

Here is a Powerpoint presentation from the NIH with instructions for reviewers: https://www.med.upenn.edu/bgs-rcr-exdes/nih-ppt.pdf

Here are some websites:

Penn PROMOTES Research on Sex and Gender in Health is a resource for sex as a biological variable and gender as a factor in differences in disease susceptibility and treatment: https://www.med.upenn.edu/penn-promotes/
NIH website on rigor and reproducibility as they relate to grants and funding: http://grants.nih.gov/reproducibility/index.htm
NIH website discussing general facets of rigor and reproducibility: https://www.nih.gov/research-training/rigor-reproducibility
NIH institute and center supported training modules being developed for use in instruction to enhance students’ abilities to conduct rigorous and reproducible research: https://www.nih.gov/research-training/rigor-reproducibility/training
NIGMS blog, Catalyzing the Modernization of Graduate Education: http://loop.nigms.nih.gov/2015/11/catalyzing-the-modernization-of-graduate-education/

Here are some meeting videocasts:

NIH has convened several meetings and workshops that are saved as videocasts. The following are links to selected topics. Please visit http://www.nih.gov/research-training/rigor-reproducibility/meetings-workshops for a complete listing.

NIH Workshop on Reproducibility in Cell Culture Studies: http://videocast.nih.gov/summary.asp?Live=16876&bhcp=1
Reproducibility of Data Collection and Analysis Modern Technologies in Cell Biology: Potentials and Pitfalls (Workshop 1 of 3): http://videocast.nih.gov/summary.asp?Live=15277&bhcp=1
Reproducibility of Data Collection and Analysis Modern Technologies in Structural Biology: Potentials and Pitfalls (Workshop 2 of 3): http://videocast.nih.gov/summary.asp?Live=15910&bhcp=1
Reproducibility of Data Collection and Analysis Modern Technologies in Structural Biology: Potentials and Pitfalls (Workshop 3 of 3): http://videocast.nih.gov/Summary.asp?File=19049&bhcp=1
The Society for Neuroscience offers several (recorded) webinars on Experimental Rigor and Enhancing Reproducibility in Neuroscience: https://neuronline.sfn.org/watch?topic={9D3303DF-4F97-4722-9F21-EECF9D239508}