University of Pennsylvania

Biomedical Graduate Studies
Responsible Conduct of Research (RCR) and Scientific Rigor and Reproducibility (SRR)

Case Study Module


The concept of reproducibility – that the same experiment always gets the same results no matter who performs it – is a cornerstone of scientific objectivity.1 In an era of increasing pace and complexity in biomedical science, however, there is concern now highlighted in several high-profile reports that reproducibility is less than that which inspires unequivocal trust from scientists and the public. While it has long be asserted that mistakes are culled as scientists move forward, i.e. that science is self-correcting, confidence in the argument is considerably less than it once was.

Problems in reproducibility are viewed to stem from problems inherent to elements of experimental design and communication of that design to the scientific community. Some contend that high on the list of factors contributing to these problems are deficits in training. Others argue more broadly. But it is important to note that problems in reproducibility are not to be confused with those that might arise through misconduct in research. It is probably for this reason, that NIH is reluctant to include reproducibility as a formal RCR topic.

NIH has issued a notice of intent to clarify and revise instructions for research grant applications, and to do the same for review criteria, to enhance reproducibility of research findings through increased scientific rigor and transparency. It has also issued a notice of a coming requirement for formal instruction in rigorous experimental design and transparency to enhance reproducibility. These notices focus on i) scientific premise, ii) rigorous experimental design, iii) consideration of sex and other relevant biological variables, and iv) authentication of key biological and/or chemical resources.

1Trouble at the lab. The Economist. October 19, 2013.



A short article by Francis Collins and Lawrence Tabak (Nature 505: 612-613, 2014) provides a good summary of problems likely underlying irreproducibility and NIH thoughts on how to surmount these problems. The link to this article is the following:


Policies and Guidelines

The University has no specific procedures or guidelines regarding reproducibility apart from those related to scientific misconduct. NIH provides guidance through a variety of notices – those of relevance are [NOT-OD-15-103] (Enhancing reproducibility through rigor and transparency) and [NOT-OD-16-034] (Advanced notice of coming requirements for formal instruction in rigorous experimental design and transparency to enhance reproducibility).

Provided here for the sake of definitions are excerpts (abridged) of a resource chart provided by the NIH relating to rigor and reproducibility in NIH applications (

Scientific Premise. The scientific premise as it relates to a grant application is the research that is used to form the basis for the proposed research question(s). The applicant should be able to describe the general strengths and weaknesses of prior research being cited as crucial to support the application. Items for discussion may well include the rigor of previous experimental designs, as well as the incorporation of relevant biological variables and authentication of key resources.

Scientific Rigor (Design). Scientific rigor is the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, and interpretation and reporting of results. In a grant application, one would emphasize how the experimental design and and methods proposed will achieve robust and unbiased results.

Biological Variables. Biological variables such as sex, age, weight, and underlying health conditions, are often critical factors affecting health or disease. In particular, sex is a biological variable that is frequently ignored in animal study designs and analyses, leading to an incomplete understanding of potential sex-based differences in basic biological function, disease processes and treatment response. In a grant application, one would explain how relevant biological variables, such as the ones noted above, are factored into research designs, analyses, and reporting in vertebrate animal and human studies.

Authentication. Key biological and/or chemical resources include, but are not limited to, cell lines, specialty chemicals, antibodies, and other biologics. In a grant application, one would describe the methods to ensure the identity and validity of key biological and/or chemical resources used in the proposed studies.


Case Studies

Access to case studies requires a PennKey. They are available here.



Resources for rigorous experimental design and transparency to enhance reproducibility (‘REDTER’) continuously evolving. We’ll do our best to keep the list up to date yet manageable.

Here are some papers:

Here is a Powerpoint presentation from the NIH with instructions for reviewers:

Here are some websites:

Here are some meeting videocasts:

NIH has convened several meetings and workshops that are saved as videocasts. The following are links to selected topics. Please visit for a complete listing.