Wednesday, October 25, 2017

In economics, 42% of papers that were published as replication studies successfully replicated another experiment

Maniadis, Z., Tufano, F. and List, J. A. (2017), To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study. Econ J, 127: F209–F235. doi:10.1111/ecoj.12527

Abstract: The sciences are in an era of an alleged ‘credibility crisis’. In this study, we discuss the reproducibility of empirical results, focusing on economics research. By combining theory and empirical evidence, we discuss the import of replication studies and whether they improve our confidence in novel findings. The theory sheds light on the importance of replications, even when replications are subject to bias. We then present a pilot meta-study of replication in experimental economics, a subfield serving as a positive benchmark for investigating the credibility of economics. Our meta-study highlights certain difficulties when applying meta-research to systematise the economics literature.

---
Concerning the replication results themselves, an interesting insight is that we found a ‘success rate’, based on 85 replications, of 42.3%. This means that roughly 40% of papers that were published as replication studies successfully replicated another experiment. This is somewhat higher than the outcome of the recent large-scale replication initiative from psychology (Open Science Collaboration, 2015) and Duvendack et al. (2015), which both found a success rate of about a third, while, also in psychology, Makel et al. (2012) found a very high success rate of 73%.

No comments:

Post a Comment