The Ideal Psychologist vs. a Messy Reality: Using and Misunderstanding Effect Sizes, Confidence Intervals and Power

Abstract

In the past two decades, there have been calls for statistical reform in psychology. Three key concepts within reform are effect sizes, confidence intervals and statistical power. The aim of this thesis was to examine the use and knowledge of these particular concepts, to examine whether researchers are suitably equipped to incorporate them into their research.

This thesis consists of five studies. Study 1 reviewed author guidelines across 100 psychology journals, to look for any statistical recommendations. Study 2 (n = 247) and Study 3 (n = 56) examined the use and knowledge of effect sizes using a questionnaire and online experiment. Study 4 surveyed psychology researchers on their use and knowledge of confidence intervals (n = 206). Similarly, Study 5 surveyed psychology researchers on their use and knowledge of power analyses and statistical power (n = 214).

Typically, psychology journals expect authors to report effect sizes in their work, although there are fewer expectations related to confidence intervals. Power analyses are also frequently encouraged for sample size justification. Self-reported use of effect sizes, confidence intervals and power analyses was high, while common barriers to use included a lack of knowledge, a lack of motivation, and the influence of academic peers. While knowledge of effect sizes was quite high, they appear to only be understood in relatively limited contexts. In contrast, both confidence intervals and statistical power appear to be frequently misunderstood, and many researchers find power analysis calculations difficult. Researchers would benefit from increased education and support to encourage them to confidently adopt an assortment of statistics in their work, and more effort must be made to prevent statistical changes from becoming a new series of tick-box exercises that do not improve the integrity of psychological research.