There are two types of error a researcher can make when conductin a null hypothesis significance testing: Type I (\(\alpha\)) and Type II (\(\beta\)) errors. Type I error is the probability of rejecting the null hypothesis, \(H_0\), when it is true. In other words, it is the false positive rate. Conventionally, type I error is set at \(\alpha = .05\). Type II error is the probability of failing to detect a significant result, when \(H_0\) is false; i.e., the false negative rate. Power is defined as \(1-\beta\) or the probability of successfully detecting an effect, when \(H_0\) is false.

There are four parameters in power analysis.

  1. Sample size
  2. Effect size
  3. Alpha level
  4. Power level

Knowing 3 of these 4, we can determine the remaining one. For example, we can conduct a priori power analysis to determine a sample size for a study with an effect size \(d = .50\), power = .80, and alpha = .05.

The current practice increasingly requires researchers to justify their sample size before conducting a study. This was done to encourage a better research practice as well as reducing wasteful resource in doing underpowered studies.

We will look at two programs for power analysis G*Power and pwr package in R.

1. G*Power

G*Power, developed by a group of researchers at Heinrich-Heine-Universität Düsseldorf, provides a user-friendly GUI for conducting power analysis. It covers wide range of statistical analysis, although some complicated design are still missing. You can download the program here.

G*Power

  1. Select the test family, e.g., t tests, F tests, \(\chi^2\) test. For this example, we will do an independent t test.
  2. Select a specific statistical test. For example, t tests can be used in many sitations: between-subject design, within-subject design, one-sample test, testing slopes or intercepts for regression. In this case, we will use “Means: Difference between two independent means (two groups)” for an independent t test.
  3. Select a type of power analysis. We will choose “A priori: Compute required sample size”
  4. Input relevant parameters
    1. Tail(s): Choose whether you use a one-tailed or two-tailed hypothesis.
    2. Effect size d: Cohen’d; hover the pointer above the box, it will show the conventional values. 4.1 The program can help you calculate d if you have information on M and SD for each group.
    3. \(\alpha\) err prob: set alpha to .05
    4. Power: set to .80
    5. Allocation ration N2/N1: If you will use balanced design (equal n), use 1.
  5. Click calculate
  6. The required sample size will appear here.

2. pwr package

#install.packages("pwr")
library(pwr)

You can read more about the package here.

The pwr package provides functions for power analysis of many statistical tests. However, for more complex design, e.g., factorial or repeated-measure, we will need to use a diferent package.

We will take a look at pwr.t.test function, which is used to analyze power for t tests with two-sample(equal n), one-sample, and paired t tests.

The function takes multiple arguments:

pwr.t.test(n = NULL, 
d = NULL, 
sig.level = 0.05, 
power = NULL, 
type = c("two.sample", "one.sample", "paired"),
alternative = c("two.sided", "less", "greater"))

To determine n, you will leave it out and enter only d and power. You also need to specify type (of t test) and alternative (two-tail or one-tail tests).

Following the same example.

pwr.t.test(d=.5, sig.level=.05, power = .8, type = "two.sample", alternative = "two.sided")
## 
##      Two-sample t test power calculation 
## 
##               n = 63.76561
##               d = 0.5
##       sig.level = 0.05
##           power = 0.8
##     alternative = two.sided
## 
## NOTE: n is number in *each* group

The n in the output tells us how many participants we need for each group.

Plotting a power graph

You can also plot the graph showing power level and sample size for a certain parameters.

medium_power <- pwr.t.test(d=.5, sig.level=.05, power = .8, type = "two.sample", alternative = "two.sided")
plot(medium_power)

Other power functions

One-way ANOVA

We can use pwr.anova.test to calculate required sample size for a particular effect size. We will need to specify k (number of groups), f (Cohen’s f), and power (desired power level).

library(pwr)
pwr.anova.test(k = 3, f = .25, power = .8) # An example of medium effect size with three conditions. 
## 
##      Balanced one-way analysis of variance power calculation 
## 
##               k = 3
##               n = 52.3966
##               f = 0.25
##       sig.level = 0.05
##           power = 0.8
## 
## NOTE: n is number in each group