The goal of any experiment is to address a specific question such that the results will be reproducible and serve as a solid foundation for further thoughts and experimental work. A solid experimental design and statistical analysis is essential for one to be able to draw correct conclusions and generate reproducible work. However, academic science programs almost never formally teach the foundations of experimental design and statistical analysis. This is incredible, really, but I have yet to see it in my 11 years in science. Experimental design and interpretation is an art form and deeply philosophical issue, but over the years people have worked out standards and fundamentals that need to be considered. I present some resources for these issues in this post.
This is a link to a well reviewed web article on experimental design by Sid Sytsma:
http://liutaiomottola.com/myth/expdesig.html
Here I also highlight two recent studies that serve as thoughtful reminders of the basics and issues of experimental design.
The first idea of interest is explained in two papers by Richter et al. in Nature Methods who argue that systematic variation improves the reproducibility of experiments compared to traditional, highly controlled standardization:
Nat Methods. 2009 Apr;6(4):257-61.
Environmental standardization: cure or cause of poor reproducibility in animal experiments?
Richter SH, Garner JP, Würbel H.
Justus-Liebig-University of Giessen, Germany.
Comment in:
Abstract
It is widely believed that environmental standardization is the best way to guarantee reproducible results in animal experiments. However, mounting evidence indicates that even subtle differences in laboratory or test conditions can lead to conflicting test outcomes. Because experimental treatments may interact with environmental conditions, experiments conducted under highly standardized conditions may reveal local ‘truths’ with little external validity. We review this hypothesis here and present a proof of principle based on data from a multilaboratory study on behavioral differences between inbred mouse strains. Our findings suggest that environmental standardization is a cause of, rather than a cure for, poor reproducibility of experimental outcomes. Environmental standardization can contribute to spurious and conflicting findings in the literature and unnecessary animal use. This conclusion calls for research into practicable and effective ways of systematic environmental heterogenization to attenuate these scientific, economic and ethical costs.
Nat Methods. 2010 Mar;7(3):167-8.
Systematic variation improves reproducibility of animal experiments.
Richter SH, Garner JP, Auer C, Kunert J, Würbel H.
Behavioural Biology, University of Münster, Münster, Germany.
The second paper by Auer and Doerge in Genetics emphasizes the importance of sampling, randomization, replication and blocking in experimental design. They deal with RNA-seq experiments specifically, but the issues are broadly relevant.
Genetics. 2010 Jun;185(2):405-16. Epub 2010 May 3.
Statistical design and analysis of RNA sequencing data.
Department of Statistics, Purdue University, West Lafayette, Indiana 47907, USA.
Abstract
Next-generation sequencing technologies are quickly becoming the preferred approach for characterizing and quantifying entire genomes. Even though data produced from these technologies are proving to be the most informative of any thus far, very little attention has been paid to fundamental design aspects of data collection and analysis, namely sampling, randomization, replication, and blocking. We discuss these concepts in an RNA sequencing framework. Using simulations we demonstrate the benefits of collecting replicated RNA sequencing data according to well known statistical designs that partition the sources of biological and technical variation. Examples of these designs and their corresponding models are presented with the goal of testing differential expression.
These are useful references for individuals in the process of designing experiments.