"The Perils of Balance Testing in Experimental Design: Messy Analyses of Clean Data." The American Statistician, 2017.

Diana C. Mutz, Robin Pemantle, and Phillip Pham

Widespread concern over the credibility of published results has led to scrutiny of statistical practices. The authors address one aspect of this problem that stems from the use of balance tests in conjunction with experimental data. When random assignment is botched, due either to mistakes in implementation or differential attrition, balance tests can be an important tool in determining whether to treat the data as observational versus experimental. Unfortunately the use of balance tests has become commonplace in analyses of “clean” data, that is, data for which random assignment can be stipulated. Here, the authors show that balance tests can destroy the basis on which scientific conclusions are formed, and can lead to erroneous and even fraudulent conclusions. They conclude by advocating that scientists and journal editors resist the use of balance tests in all analyses of clean data.

Published online in June 2017