There are two types of error a researcher can make when conductin a null hypothesis significance testing: Type I ($$\alpha$$) and Type II ($$\beta$$) errors. Type I error is the probability of rejecting the null hypothesis, $$H_0$$, when it is true. In other words, it is the false positive rate. Conventionally, type I error is set at $$\alpha = .05$$. Type II error is the probability of failing to detect a significant result, when $$H_0$$ is false; i.e., the false negative rate. Power is defined as $$1-\beta$$ or the probability of successfully detecting an effect, when $$H_0$$ is false.

There are four parameters in power analysis.

1. Sample size
2. Effect size
3. Alpha level
4. Power level

Knowing 3 of these 4, we can determine the remaining one. For example, we can conduct a priori power analysis to determine a sample size for a study with an effect size $$d = .50$$, power = .80, and alpha = .05.

The current practice increasingly requires researchers to justify their sample size before conducting a study. This was done to encourage a better research practice as well as reducing wasteful resource in doing underpowered studies.

We will look at two programs for power analysis G*Power and pwr package in R.

# 1. G*Power

G*Power, developed by a group of researchers at Heinrich-Heine-Universität Düsseldorf, provides a user-friendly GUI for conducting power analysis. It covers wide range of statistical analysis, although some complicated design are still missing. You can download the program here.

1. Select the test family, e.g., t tests, F tests, $$\chi^2$$ test. For this example, we will do an independent t test.
2. Select a specific statistical test. For example, t tests can be used in many sitations: between-subject design, within-subject design, one-sample test, testing slopes or intercepts for regression. In this case, we will use “Means: Difference between two independent means (two groups)” for an independent t test.
3. Select a type of power analysis. We will choose “A priori: Compute required sample size”
4. Input relevant parameters
1. Tail(s): Choose whether you use a one-tailed or two-tailed hypothesis.
2. Effect size d: Cohen’d; hover the pointer above the box, it will show the conventional values. 4.1 The program can help you calculate d if you have information on M and SD for each group.
3. $$\alpha$$ err prob: set alpha to .05
4. Power: set to .80
5. Allocation ration N2/N1: If you will use balanced design (equal n), use 1.
5. Click calculate
6. The required sample size will appear here.

# 2. pwr package

#install.packages("pwr")
library(pwr)

The pwr package provides functions for power analysis of many statistical tests. However, for more complex design, e.g., factorial or repeated-measure, we will need to use a diferent package.

We will take a look at pwr.t.test function, which is used to analyze power for t tests with two-sample(equal n), one-sample, and paired t tests.

The function takes multiple arguments:

pwr.t.test(n = NULL,
d = NULL,
sig.level = 0.05,
power = NULL,
type = c("two.sample", "one.sample", "paired"),
alternative = c("two.sided", "less", "greater"))

To determine n, you will leave it out and enter only d and power. You also need to specify type (of t test) and alternative (two-tail or one-tail tests).

Following the same example.

pwr.t.test(d=.5, sig.level=.05, power = .8, type = "two.sample", alternative = "two.sided")
##
##      Two-sample t test power calculation
##
##               n = 63.76561
##               d = 0.5
##       sig.level = 0.05
##           power = 0.8
##     alternative = two.sided
##
## NOTE: n is number in *each* group

The n in the output tells us how many participants we need for each group.

# Plotting a power graph

You can also plot the graph showing power level and sample size for a certain parameters.

medium_power <- pwr.t.test(d=.5, sig.level=.05, power = .8, type = "two.sample", alternative = "two.sided")
plot(medium_power)

# Other power functions

## One-way ANOVA

We can use pwr.anova.test to calculate required sample size for a particular effect size. We will need to specify k (number of groups), f (Cohen’s f), and power (desired power level).

library(pwr)
pwr.anova.test(k = 3, f = .25, power = .8) # An example of medium effect size with three conditions. 
##
##      Balanced one-way analysis of variance power calculation
##
##               k = 3
##               n = 52.3966
##               f = 0.25
##       sig.level = 0.05
##           power = 0.8
##
## NOTE: n is number in each group