Tuesday, June 15, 2021

Psychological research: Guidance for systematically evaluating whether an observed effect size is practically relevant is needed to justify claims about practical relevance; that is, there is no such mechanism as of now

Anvari, Farid, Rogier Kievit, Daniel Lakens, Andrew K. Przybylski, Leonid Tiokhin, Brenton M. Wiernik, and Amy Orben. 2021. “Evaluating the Practical Relevance of Observed Effect Sizes in Psychological Research.” PsyArXiv. June 15. doi:10.31234/osf.io/g3vtr

Abstract: Psychological researchers currently lack guidance for how to evaluate the practical relevance of observed effect sizes, i.e. whether a finding will have impact when translated to a different context of application. Although psychologists have recently highlighted theoretical justifications for why small effect sizes might be practically relevant, such justifications are simplistic and fail to provide the information necessary for evaluation and falsification. Claims about whether an observed effect size is practically relevant need to consider both the mechanisms amplifying and counteracting practical relevance, as well as the assumptions underlying each mechanism at play. To provide guidance for systematically evaluating whether an observed effect size is practically relevant, we present examples of widely applicable mechanisms and the key assumptions needed for justifying whether an observed effect size can be expected to generalize to different contexts. Routine use of these mechanisms to justify claims about practical relevance has the potential to make researchers’ claims about generalizability substantially more transparent. This transparency can help move psychological science towards a more rigorous assessment of when psychological findings can be applied in the world.

Update Jul 5 2021: Collins, Elizabeth, and Roger Watt. 2021. “Use, Knowledge and Misconceptions of Effect Sizes in Psychology.” PsyArXiv. July 1. doi:10.31234/osf.io/r7vmf

Abstract: In this study, 247 psychologists were surveyed to examine effect size use, barriers to use, and effect size knowledge measured using quantitative and qualitative questions. This pre-print reports our findings.

No comments:

Post a Comment