A study of transcranial electromagnetic stimulation, published In

Another way to express the strength of the evidence provided by P = 0.043 is to note that it makes the existence of a real effect only 3.3 times as likely as the existence of no effect (likelihood ratio). This would correspond to a minimum false positive risk of 23% if we were willing to assume that non-specific electrical zapping of the brain was as likely as not to improve memory (prior probability of a real effect was 0.5) (found via radio button 3).

The radio button 2 option shows that in the most optimistic case (prior = 0.5), you need to have P = 0.008 to achieve an FPR of 5 percent. (Example from refs [3] and [7].)

For example effect size = 1 and n = 16 gives power = 0.78. For an effect size of 0.5 SD, n = 61 gives similar power and also a similar FPR etc. And for an effect size of 0.2 SD, a power of 0.78 requires n = 375, and again this gives similar FPR etc. See ref [7] for more details. So choose n so that the calculated power matches that of your experiment. There is a popular account of the logic involved in ref [4]. And ref [3] has, in section 9, a response to the recent 72 author paper, Benjamin et al [5], on related topics. There is a more technical account of the assumptions in ref [7]. And a defence of those assumptions in ref [8].

2. Colquhoun D. False discovery rates: the movie (now superseded by ref [6]) Click for YouTube

3. Colquhoun D. (2017). The reproducibility of research and misinterpretation of P values.

4. Colquhoun D. (2016). The problem with p-values. Aeon Magazine Click for full text

5. Benjamin, D. et al. (2017) Redefine Statistical Significance. PsyArXiv Preprints, July 22, 2017. Click for full text

6. Colquhoun, D. (2018) Colquhoun D. The false positive risk: a proposal concerning what to do about p-values (version 2) [talk based on that given at EvidenceLive, 2018] Click for YouTube

7. Colquhoun, D. (2019) The false positive risk: a proposal concerning what to do about p values.

8. Colquhoun, D. (2019b) A response to critiques of "The reproducibility of research and the misinterpretation of p-values".

A list of all of DC's publications on p values can be found at Some papers about p values.