This Shiny app accompanies the paper 'Sample Size Justification' by Daniël Lakens. You can download the pre-print of this article at PsyArXiV and any sections in this online form that are unclear are explained in the paper. You can help to improve this app by providing feedback or suggest additions by filling out this feedback form . Note that this app will not store the information you enter if you close or refresh you browser. You might want to write down answers in a local text file first. For a completed example, see here .

The main goal of this app and the accompanying paper is to guide you through an evaluation of the informational value of a planned study. After filling out this form you can download a report of your sample size justification.

More Info on the Informational Value of Studies.



The informational value of a study depends on the inferential goal, which could be testing a hypothesis, obtaining an accurate estimate, or seeing what you can learn from all the data you have the resources to collect.

  • It is possible that your resource constraints allow you to perform a study that has:
    • a desired statistical power, or
    • a desired accuracy of the estimated effect
    and your resource constraints are not the primary reason to collect a specific sample size (even though resource constraints are always a secondary reason to collect a certain sample size, as without resource constraints, one would for example choose a very low alpha level and design a study that has incredibly high statistical power). In these cases, you would:
    • perform an a-priori power analysis for the smallest effect sizes of interest or, if that can not be specified, for an a-priori power analysis for an expected effect size.
    • determine the sample size using an accuracy in parameter estimates perspective, based on the desired accuracy and the expected effect size.

  • It is also possible that the calculations based on power and accuracy yield a sample size that is larger than you have the resources to collect. In these situations, you can:
    • not draw any inferences, and collect the data so they can be included in a future meta-analysis.
    • justify the sample size because a decision needs to be made, even if data is scarce, and design a study based on a compromise power analysis that allows you to sufficiently reduce the relative probability of Type 1 and Type II error rates based on a cost-benefit analysis
    • If you still want to perform a hypothesis test, perform a sensitivity power analysis, justify the sample size based on the information it will provide about the expected effect size or other effect sizes of interest, such as effects previously observed in the literature. If you plan to perform a hypothesis test, examine if the minimal statistically detectable effect is small enough to warrant a hypothesis test, and evaluate whether the Type 1 error rate and the Type II error rate make it possible to draw useful conclusions based on the p-value, or not.
    • If you want to estimate an effect size, interpret the width of the confidence interval around the estimate, and specify what an estimate with this accuracy is useful for.

Describe Your Population

Describe the population and its size

Describe the population you are sampling from.

Can you collect data from the entire population?

Describe your resource constraints.

Describe your resource constraints (e.g., time and money), and how these limit the maximum sample size you are willing and able to collect.

Which Effect Sizes are of Interest?



A shared feature of the different inferential goals (see Part C) is the question which effect sizes a researcher considers meaningful to learn about. This implies that researchers need to evaluate which effect sizes they consider interesting. First, it is useful to consider three effect sizes when determining the sample size. The first is the smallest effect size a researcher is interested in, the second is the smallest effect size that can be statistically significant (only in studies where a significance test will be performed), and the third is the effect size that is expected. Below, you can provide details about effect sizes of interest (but you can also leave all fields empty).

Smallest Effect Size of Interest

What is the smallest effect size of interest (specify the metric and the value)? What is the justification to consider this the smallest effect size of interest? For example, is this the smallest effect that would be practically relevant, theoretically predicted, or that would reject a previously observed effect using the Small Telescopes approach?

For examples, see here .

Minimal Statistically Detectable Effect

What is the minimal statistically detectable effect? How was the minimal statistically detectable effect computed (preferably in code)?

Expected Effect Size

What is the expected effect size, and why?

What is the source of the expected effect size? E.g., a meta-analysis, a previous study, or a subjective prior belief. If applicable, cite the source, and add a direct quote or table number that contains the effect size estimate.

Can the effect size from the source be expected to generalize to the planned study? For a meta-analyses with large heterogeneity, what is the effect size in the most heterogenous subset?

Is there a risk of bias in the effect size estimate, and if relevant, is the source effect size adjusted in any way?

Below, choose either the options that the expected effect size is based on a meta-analysis, or choose the option that the expected effect size is based on a previous study, or specify what the source of the expectation was in the main text field.

Is the expected effect size based on a meta-analysis?

Recommendations for effect size estimates from a meta-analysis.

For examples, see here .

Are the studies in the meta-analysis similar to the study you are planning? Evaluate the generalizability of the effect size in the meta-analysis to your study. Include a citation to the meta-analysis, and if possible copy-paste text from the meta-analysis that reports the effect size.

Are the studies in the meta-analysis homogeneous? If there is a lot of heterogeneity, which subsample of studies would be most relevant to the study you are planning?

Is the meta-analytic effect size estimate unbiased? If not, can you compute an effect size estimate that attempts to correct for bias, or will you use a more conservative effect size estimate?

Is the expected effect size based on a previous study?

Recommendations for effect size estimates from a single study.

For examples, see here .

Is the previous study similar to the study you are planning? Evaluate the generalizability of the effect size in the previous study to your study. Include a citation to the study, and if possible copy-paste text from the original study that reports the effect size.

How large is the uncertainty in the effect size estimate of the previous study? How have you dealt with this uncertainty (e.g., choosing a more conservative effect size)?

Is the effect size estimate unbiased? If not, can you compute an effect size estimate that attempts to correct for bias, or will you use a more conservative effect size estimate?

Width of the Confidence Interval

If a researcher can estimate the standard deviation of the observations that will be collected, it is possible to compute an a-priori estimate of the width of the confidence interval around an effect size. What is the informational value of estimating effects with this accuracy? Which effect sizes can be expected to be rejected, and why is it interesting for peers?

Sensitivity Power Analysis

Across a range of possible effect sizes, which effects does a design have sufficient power to detect when performing a hypothesis test?

Distribution of Effect Sizes

What is the distribution of effect sizes in this research area? Add a citation to a meta-meta-analysis, where possible.

Specify the Inferential Goal.

By collecting a certain amount of data, researchers aim to reach an inferential goal. Common inferential goals are to make a decision, perform a statistical test, or measure an effect with a desired accuracy. It is also possible that data collection has no specified inferential goal, either because data is only collected to provide input for a future meta-analysis, because a sample size is based on heuristics, or because there is no justification for the sample size.

Input for Future Meta-Analysis

Will this study mainly serve as input for a future meta-analysis (and will no inferences be drawn from this dataset in isolation?

How will the meta-analysis be realized?

Reflect on the probability that a future meta-analysis will be performed, how you will help to realize such a meta-analysis.

Decision

Is there a clear need to make a decision about how to act based on the results of this study?

Specify cost/benefit considerations

For examples, see here .

Specify the parameters of the cost/benefit trade-off used in the power analysis.

Estimation

Is your inferential goal to estimate the size of a parameter?

Estimation details.

Specify the parameters related to the desired estimation accuracy.

Statistical Power

Is your inferential goal to achieve a desired statistical power for a statistical test?

For examples of sensitivity power analyses, see here .

Power calculation.

Specify the parameters related to the statistical power calculation. If you have multiple hypothesis tests in your study, enter the power analysis that determines your final sample size. Typically, this is the power analysis that returns the largest number of observations.

Heuristic

Is the sample size based on a heuristic?

Heuristic details.

It is important to know what the logic behind a heuristic is to determine whether the heuristic is valid for a specific situation. Try to identify the source of the heuristic, and describe the logic of the heuristic. In most cases, heuristics are not tied to any specified inferential goal, and therefore there is a risk their use to determine the sample size leads to studies that lack informational value.

No Justification

Is the sample size determined without any justification in mind?

No Justification details.

It is useful to distinguish a final category where researchers explicitly state they do not have a justification for their sample size. In those cases, instead of pretending there was a justification for the sample size, honesty requires you to state there is no sample size justification, and evaluate which effect sizes of interest the study could provide information about.

Based on the resource constraints, the effects of interest, and the inferential goals, specify the sample size you plan to collect, and evaluate the informational value of the study.

Total number of participants

Total observations per participant

Additional details about the sample size

Describe the distribution of participants or observations across conditions, how you plan to deal with missing data, or any other information that determines the information this data can provide in relation to the inferential goal.

Given the following resource constraints:

Given the following effect(s) of interest:

Given the following inferential goal(s):

Please explain what the the informational value of the sample size that will be collected is, given any resource constraints, the effects of interest, and the inferential goal.

Informational Value of the Study



You can download a html report of your sample size justification (for example to add it to a preregistration of your study) by clicking the button below.

Download Sample Size Justification