Tutorial 9 - One-way ANOVA

Before you start, make sure the ez package is installed. If library(ez) gives you an error, run this once in the console:

install.packages("ez")

# load libraries

library(data.table)
library(ggplot2)
library(ez)

# clean work space

rm(list = ls())

# init colorscheme

COL <- c("#2271B2", "#E69F00", "#D55E00")
names(COL) <- c("blue", "orange", "red")
theme_set(
  theme_minimal(base_size = 13) +
    theme(
      panel.grid.minor = element_blank(),
      strip.text = element_text(face = "bold"),
      legend.position = "bottom"
    )
)
update_geom_defaults("point", list(size = 2))
update_geom_defaults("line", list(linewidth = 0.8))

Overview

So far, you have used t-tests to compare two means. In this tutorial, you will learn what to do when you have three or more independent groups.

If you ran every possible pairwise t-test, your chance of making at least one false positive, also called a Type I error, would increase across the set of tests. One-way ANOVA solves this problem by testing all group means together in a single omnibus F-test.

In this tutorial, you will work with a simulated attention-training dataset in which different participants completed one of three training schedules:

brief
moderate
extended

Your dependent variable will be mean response time (mean_rt). Each row in the dataset is one participant, and each participant appears in only one training condition. That makes this a true one-factor, between-subjects design.

Part 1 - Introducing the data

d <- fread("data/attention_training_summary.csv")
str(d)
## Classes 'data.table' and 'data.frame':   36 obs. of  4 variables:
##  $ participant_id    : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ training_condition: chr  "brief" "brief" "brief" "brief" ...
##  $ mean_rt           : num  712 690 702 725 694 ...
##  $ prop_correct      : num  0.79 0.82 0.8 0.78 0.81 0.77 0.8 0.82 0.78 0.79 ...
##  - attr(*, ".internal.selfref")=<externalptr>
head(d)
##    participant_id training_condition mean_rt prop_correct
##             <int>             <char>   <num>        <num>
## 1:              1              brief   712.4         0.79
## 2:              2              brief   689.7         0.82
## 3:              3              brief   701.9         0.80
## 4:              4              brief   725.3         0.78
## 5:              5              brief   694.5         0.81
## 6:              6              brief   718.2         0.77

This dataset contains one row per participant and four variables:

participant_id - participant identifier
training_condition - the training schedule assigned to that participant
mean_rt - the participant’s mean response time in milliseconds
prop_correct - the participant’s mean proportion correct

Before doing any inferential test, always check the unit of analysis. The unit of analysis is the thing that each row represents in the dataset you give to the test. For this ANOVA, each row is one participant, not one trial. That matters because a between-subjects one-way ANOVA assumes that the rows are independent observations. If we used every trial as a separate row, the same participant would appear many times and those rows would not be independent.

Here, the unit of analysis is the participant. Each participant contributes one summary score (mean_rt) and belongs to one training condition. That is the correct level for a between-subjects one-way ANOVA.

For the rest of this tutorial, focus on response time.

d[, training_condition := factor(
  training_condition,
  levels = c("brief", "moderate", "extended")
)]
d[, participant_id := factor(participant_id)]

Part 2 - Visualise the group differences

Start with a boxplot.

ggplot(d, aes(x = training_condition, y = mean_rt, fill = training_condition)) +
  geom_boxplot(outlier.shape = NA) +
  geom_jitter(width = 0.12, alpha = 0.5) +
  scale_fill_manual(values = COL) +
  labs(
    x = "Training condition",
    y = "Mean response time (ms)",
    title = "Response time by attention-training condition",
    fill = "Condition"
  ) +
  theme(legend.position = "none")

Mean response time by training condition.

Now compute the group means and standard errors.

d_sum <- d[, .(
  mean_rt = mean(mean_rt),
  se_rt   = sd(mean_rt) / sqrt(.N)
), by = training_condition]

d_sum
##    training_condition  mean_rt    se_rt
##                <fctr>    <num>    <num>
## 1:              brief 707.2667 3.423235
## 2:           moderate 652.9833 2.484980
## 3:           extended 596.3333 2.177374

ggplot(d_sum, aes(x = training_condition, y = mean_rt, fill = training_condition)) +
  geom_col(width = 0.6) +
  geom_errorbar(
    aes(ymin = mean_rt - se_rt, ymax = mean_rt + se_rt),
    width = 0.15
  ) +
  scale_fill_manual(values = COL) +
  labs(
    x = "Training condition",
    y = "Mean response time (ms)",
    title = "Mean response time by condition",
    fill = "Condition"
  ) +
  theme(legend.position = "none")

Group means with standard error bars.

From the plots, it looks as though response time decreases as training becomes more extensive. The next question is whether those differences are large enough to be statistically reliable.

Part 3 - Why not just run three t-tests?

With three groups, there are three pairwise comparisons:

brief vs moderate
brief vs extended
moderate vs extended

You could run three independent-samples t-tests:

t.test(mean_rt ~ training_condition,
       data = d[training_condition %in% c("brief", "moderate")],
       var.equal = TRUE)
## 
##  Two Sample t-test
## 
## data:  mean_rt by training_condition
## t = 12.833, df = 22, p-value = 1.085e-11
## alternative hypothesis: true difference in means between group brief and group moderate is not equal to 0
## 95 percent confidence interval:
##  45.51066 63.05600
## sample estimates:
##    mean in group brief mean in group moderate 
##               707.2667               652.9833

t.test(mean_rt ~ training_condition,
       data = d[training_condition %in% c("brief", "extended")],
       var.equal = TRUE)
## 
##  Two Sample t-test
## 
## data:  mean_rt by training_condition
## t = 27.343, df = 22, p-value < 2.2e-16
## alternative hypothesis: true difference in means between group brief and group extended is not equal to 0
## 95 percent confidence interval:
##  102.5196 119.3471
## sample estimates:
##    mean in group brief mean in group extended 
##               707.2667               596.3333

t.test(mean_rt ~ training_condition,
       data = d[training_condition %in% c("moderate", "extended")],
       var.equal = TRUE)
## 
##  Two Sample t-test
## 
## data:  mean_rt by training_condition
## t = 17.146, df = 22, p-value = 3.239e-14
## alternative hypothesis: true difference in means between group moderate and group extended is not equal to 0
## 95 percent confidence interval:
##  49.79803 63.50197
## sample estimates:
## mean in group moderate mean in group extended 
##               652.9833               596.3333

A false positive is a Type I error: you conclude that there is a difference when, in reality, the null hypothesis is true. If you use an alpha level of .05, then one test has a 5% chance of producing a false positive when there is no real effect.

The problem is that this false-positive risk accumulates across a set of tests. If you run three separate t-tests, you have three chances to make a false positive error. Even if every null hypothesis is true, the chance that at least one test gives p < .05 is larger than 5%.

If the tests were independent, the probability of at least one false positive across three tests would be:

\[1 - (1 - .05)^3 = .143\]

That is about 14%, not 5%.

Here is the same idea using a simulation. In this simulation, we run three independent t-tests where all samples come from exactly the same population, so every “significant” result is a false positive.

set.seed(2026)

n_sim <- 5000
alpha <- .05
n_per_group <- 12

any_false_positive <- replicate(n_sim, {
  p_values <- c(
    t.test(rnorm(n_per_group), rnorm(n_per_group), var.equal = TRUE)$p.value,
    t.test(rnorm(n_per_group), rnorm(n_per_group), var.equal = TRUE)$p.value,
    t.test(rnorm(n_per_group), rnorm(n_per_group), var.equal = TRUE)$p.value
  )

  any(p_values < alpha)
})

mean(any_false_positive)
## [1] 0.141

The exact answer changes slightly from simulation to simulation, but it should be close to .143 and clearly bigger than .05. One-way ANOVA avoids starting with three separate tests by giving you a single omnibus test of:

\[H_0: \mu_{brief} = \mu_{moderate} = \mu_{extended}\]

Planned comparisons are a special case. If you decide before looking at the data that you only care about one specific comparison, that test is not the same as running every possible pairwise comparison and searching for significance. But if you plan several comparisons, the false-positive risk still accumulates across that planned set, so some form of correction may still be appropriate. The key distinction is whether the comparison was specified in advance or chosen after inspecting the results.

If the omnibus test is significant, you then follow it up with corrected post-hoc comparisons.

Part 4 - Run the one-way ANOVA

In the lectures, we use ezANOVA() from the ez package. For a one-way between-subjects ANOVA, we specify:

data - the data table containing the data
dv - the dependent variable
wid - the participant identifier
within = NULL - there are no within-subjects factors in this design
between - the between-subjects factor
type = 3 - Type III sums of squares, which we will use throughout ANOVA

fit <- ezANOVA(
  data       = d,
  dv         = mean_rt,
  wid        = .(participant_id),
  within     = NULL,
  between    = .(training_condition),
  type       = 3,
  return_aov = TRUE
)

fit$ANOVA
##               Effect DFn DFd        F           p p<.05       ges
## 2 training_condition   2  33 407.8289 5.38755e-24     * 0.9611151
fit$`Levene's Test for Homogeneity of Variance`
##   DFn DFd      SSn      SSd        F         p p<.05
## 1   2  33 88.61556 725.6933 2.014841 0.1494188

How to read the output:

Effect is the factor being tested
DFn is the numerator degrees of freedom for the effect
DFd is the denominator degrees of freedom for the error term
F is the ANOVA test statistic
p is the p-value for the omnibus test
p<.05 marks whether the result is significant at the .05 level
ges is generalised eta-squared, a measure of effect size

If the p-value is small, you can conclude that not all three group means are equal.

Part 5 - Post-hoc comparisons

If the omnibus ANOVA is significant, you still need to ask which groups differ. Tukey’s Honestly Significant Difference test is a standard choice for all pairwise comparisons in a one-way ANOVA.

Because we used return_aov = TRUE above, the ezANOVA() result includes the underlying aov object. We can use that object for Tukey’s test.

TukeyHSD(fit$aov)
##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = formula(aov_formula), data = data)
## 
## $training_condition
##                         diff        lwr        upr p adj
## moderate-brief     -54.28333  -63.81523  -44.75144     0
## extended-brief    -110.93333 -120.46523 -101.40144     0
## extended-moderate  -56.65000  -66.18189  -47.11811     0

This output gives the mean difference for each pair, a confidence interval, and an adjusted p-value that controls the family-wise error rate.

Sometimes R prints very small p-values as 0. This does not mean the probability is exactly zero. Report these as very small, for example p < .001.

Interpretation guide:

if the adjusted p-value is below your alpha level, that pair differs
if the confidence interval does not include zero, that is consistent with a reliable difference

Part 6 - Assumptions

One-way ANOVA depends on four main assumptions:

Independence - each participant contributes one score and participants are independent of one another. There should be no repeated measures unless you are using a repeated-measures ANOVA.
Normality - residual scores are approximately normally distributed. Residuals are the differences between observed scores and their group means.
Homogeneity of variance - the population variances are equal across groups. This is also called homoscedasticity.
Scale of measurement - the dependent variable is continuous, and the independent variable is categorical.

Normality of residuals

The lectures show how to extract residuals from ezANOVA() using return_aov = TRUE. Once you have the residuals, you can check their distribution visually. Here we use a Q-Q plot as another visual check of normality.

d_resid <- data.table(resid = residuals(fit$aov))

ggplot(d_resid, aes(sample = resid)) +
  stat_qq(colour = COL[1]) +
  stat_qq_line(colour = COL[2]) +
  labs(
    x = "Theoretical quantiles",
    y = "Sample quantiles",
    title = "Q-Q plot of residuals"
  )

Q-Q plot of ANOVA residuals.

Homogeneity of variance

Levene’s test is a common test for homogeneity of variance, and it is provided in the ezANOVA() output.

fit$`Levene's Test for Homogeneity of Variance`
##   DFn DFd      SSn      SSd        F         p p<.05
## 1   2  33 88.61556 725.6933 2.014841 0.1494188

Use these checks to support your interpretation, not to replace it. With moderate group sizes, ANOVA is reasonably robust to mild deviations, but you should still inspect the assumptions rather than ignore them.

Apply this to your assigned dataset

Your assigned dataset is determined by your student ID number. Take the last digit and compute last_digit %% 3 in R.

If the result is 0: https://github.com/crossley/cogs2020/tree/main/final_project_data/cat_learn_switch
If the result is 1: https://github.com/crossley/cogs2020/tree/main/final_project_data/cat_learn_auto
If the result is 2: https://github.com/crossley/cogs2020/tree/main/final_project_data/cat_learn_unlearn

Before you start, check whether your assigned dataset can support an independent-groups question this week. All of the final project datasets include at least one within-subjects factor, so everyone needs to be careful about independence. If the same participant contributes scores across multiple conditions, blocks, or sessions, those rows are repeated measures and should not be treated as independent groups in a between-subjects one-way ANOVA. Do not use a within-subjects factor as the one-way ANOVA factor this week. You may need to choose a genuinely between-subjects question, aggregate to one independent score per participant or unit, or wait for the repeated-measures ANOVA in the next tutorial.

If your question uses independent groups, load your assigned dataset and run a one-way ANOVA using ezANOVA(). Specifically:

Identify one independent variable with three or more levels.
Make sure your rows are independent at the level you analyse.
Aggregate your data to one score per participant or unit if needed.
Run a one-way ANOVA with ezANOVA().
Report the F statistic, degrees of freedom, and p-value.
If the result is significant, run post-hoc comparisons.
Create at least one plot showing the group differences.
Write a short paragraph interpreting the result in plain language.