There are two types of error a researcher can make when conductin a null hypothesis significance testing: Type I (\(\alpha\)) and Type II (\(\beta\)) errors. Type I error is the probability of rejecting the null hypothesis, \(H_0\), when it is true. In other words, it is the false positive rate. Conventionally, type I error is set at \(\alpha = .05\). Type II error is the probability of failing to detect a significant result, when \(H_0\) is false; i.e., the false negative rate. Power is defined as \(1-\beta\) or the probability of successfully detecting an effect, when \(H_0\) is false.
There are four parameters in power analysis.
Knowing 3 of these 4, we can determine the remaining one. For example, we can conduct a priori power analysis to determine a sample size for a study with an effect size \(d = .50\), power = .80, and alpha = .05.
The current practice increasingly requires researchers to justify their sample size before conducting a study. This was done to encourage a better research practice as well as reducing wasteful resource in doing underpowered studies.
We will look at two programs for power analysis G*Power and
pwr package in R.
G*Power, developed by a group of researchers at Heinrich-Heine-Universität Düsseldorf, provides a user-friendly GUI for conducting power analysis. It covers wide range of statistical analysis, although some complicated design are still missing. You can download the program here.
You can read more about the package here.
pwr package provides functions for power analysis of many statistical tests. However, for more complex design, e.g., factorial or repeated-measure, we will need to use a diferent package.
We will take a look at
pwr.t.test function, which is used to analyze power for t tests with two-sample(equal n), one-sample, and paired t tests.
The function takes multiple arguments:
pwr.t.test(n = NULL, d = NULL, sig.level = 0.05, power = NULL, type = c("two.sample", "one.sample", "paired"), alternative = c("two.sided", "less", "greater"))
n, you will leave it out and enter only
power. You also need to specify
type (of t test) and
alternative (two-tail or one-tail tests).
Following the same example.
pwr.t.test(d=.5, sig.level=.05, power = .8, type = "two.sample", alternative = "two.sided")
## ## Two-sample t test power calculation ## ## n = 63.76561 ## d = 0.5 ## sig.level = 0.05 ## power = 0.8 ## alternative = two.sided ## ## NOTE: n is number in *each* group
The n in the output tells us how many participants we need for each group.
You can also plot the graph showing power level and sample size for a certain parameters.
medium_power <- pwr.t.test(d=.5, sig.level=.05, power = .8, type = "two.sample", alternative = "two.sided") plot(medium_power)
We can use
pwr.anova.test to calculate required sample size for a particular effect size. We will need to specify
k (number of groups),
f (Cohen’s f), and
power (desired power level).
library(pwr) pwr.anova.test(k = 3, f = .25, power = .8) # An example of medium effect size with three conditions.
## ## Balanced one-way analysis of variance power calculation ## ## k = 3 ## n = 52.3966 ## f = 0.25 ## sig.level = 0.05 ## power = 0.8 ## ## NOTE: n is number in each group