Thursday, February 16, 2017

Rigor and Reproducibility

Rigor and Reproducibility

A recent report found that the published findings from 89% of preclinical studies in the field of cancer research – in particular, those deemed ‘landmark’ studies – were unable to be confirmed by other scientists [1]. The authors concluded that despite competent, well-intentioned researchers, an alarming number of these inconsistencies were due to limitations in the design and execution of the initial studies, including the use of problematic endpoint measures and biased analysis of data.

This report is not unique to one particular research field, but is in fact representative of a growing awareness within the greater scientific community of the critical need for robust and unbiased experimental design [2-5]. The National Institutes of Health (NIH), one of the world’s leading medical research centers and major provider of extramural biomedical research funds, recently implemented new grant application instructions intended to enhance reproducibility through rigor and transparency [6]. Research grant applicants must now address four areas of focus:  scientific premise, scientific rigor, biological variables, and authentication.

The NIH defines scientific rigor as “the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results.” This standard has been particularly difficult to achieve for imaging-based studies, which conventionally depend heavily on manual methods for collecting and analyzing data. This time-consuming approach frequently offers limited quantitative results due to small sample size and qualitative endpoints. Additionally, subjective selection criteria and lack of standardization across methods and instrumentation impedes reproducibility and transparency.

Recent advances in automated imaging systems have provided a great leap forward in our ability to conduct robust imaging-based biomedical research. The efficient and consistent acquisition of high quality images by BioTek’s Lionheart FX and Cytation automated imagers, across large numbers of conditions and replicates, allows researchers to meet the high standards set for reproducibility and transparency. Furthermore, the routine application of powerful Gen5 processing and analysis tools across entire image sets enables robust and meaningful quantitative results. 

Access to innovative technology and the prospect of life-improving therapeutics on the horizon make this an exciting time to be a scientist. Successfully translating biomedical research into clinical advances requires rigorous and reproducible approaches.  In our rapidly evolving scientific landscape, the need to demonstrate the relevance and benefit of sound scientific research is as important now as ever before.

  1. Begley, C.G. and L.M. Ellis, Drug development: Raise standards for preclinical cancer research. Nature, 2012. 483(7391): p. 531-3.
  2. Begley, C.G., A.M. Buchan, and U. Dirnagl, Robust research: Institutions must do their part for reproducibility. Nature, 2015. 525(7567): p. 25-7.
  3. Begley, C.G. and J.P. Ioannidis, Reproducibility in science: improving the standard for basic and preclinical research. Circ Res, 2015. 116(1): p. 116-26.
  4. Kretser, A., D. Murphy, and J. Dwyer, Scientific integrity resource guide: Efforts by federal agencies, foundations, nonprofit organizations, professional societies, and academia in the United States. Crit Rev Food Sci Nutr, 2017. 57(1): p. 163-180.
  5. Pusztai, L., C. Hatzis, and F. Andre, Reproducibility of research and preclinical validation: problems and solutions. Nat Rev Clin Oncol, 2013. 10(12): p. 720-4.
  6. Health, N.I.o. 2017.

By, BioTek Instruments, Joe Clayton, PhD., Principal Scientist

No comments:

Post a Comment