Hypothesis testing
Significance testing

We are learning how to think critically for ourselves

Active learning works best even if it sometimes feel challenging

• Active learning is what you do when you ask or answer questions
• We will prompt you sometimes with Activate

We are going to learn about hypothesis or significance testing

This is a challenging topic

• We can meet this challenge by enriching our perspectives

  • historical, philosophical, psychological and statistical

• This approach contrasts with a focus on procedure

  • Here, we aim to understand the ideas better so we can apply them confidently

What are we talking about?

• What we are talking about is usually called Null Hypothesis Significance Testing (NHST)

What we do when we do NHST is counter-intuitive

When we work with null hypothesis significance tests

• We are often working in ways that we find difficult

• This is because these ways appear to run counter to our intuitions(Gigerenzer, 1993, 2004; Tversky & Kahneman, 1971)

• the habitual or instinctive ways of thinking we rely on in everyday life

Students, researchers, teachers and textbooks make mistakes

• Researchers often make mistakes in reporting NHST results (Bakker & Wicherts, 2011; Nuijten et al., 2016; Nuijten & Wicherts, 2024):

  - Inconsistent or incomplete reports

• Students, teachers and textbooks often make mistakes in explaining significance testing methods (Cassidy et al., 2019; Gigerenzer, 2018; Haller & Kraus, 2002; Murphy et al., 2025):

  - Mixing up ideas, applying incorrect interpretations

• Why do we think we can do better?

NHST is everywhere and it is clearly important

• We may say people often get NHST wrong
• But psychologists will spend most of their time doing and reporting hypothesis or significance tests (Cumming et al., 2007; Nuijten et al., 2016; Nuijten & Wicherts, 2024; Scheel et al., 2021)
• So this is a thing we have to learn about

This feels like a scary situation

• Gigerenzer (1993) suggests that, because we get anxious about it, we (psychologists) adopt ritualistic behaviour:

  • Fixate on special numbers: .05
  • Repeat the same action
  • Fear being punished

• This idea connects to the sense that the challenge here is psychological: it’s about the psychology of psychologists

OK then, how do we meet the challenge?

Step One: What is NHST and where does it come from?

• We are going to look at ideas from history, philosophy and psychology here
• It is a fascinating story and we will see that knowing about it helps to anchor our understanding

Step Two: Seeing is believing

• Later, we will grow our understanding by looking at practical applications of the key ideas
• We will focus here on demonstrations of how we apply significance tests or hypothesis tests (properly understood) in the situations psycholgists face every day

Step Three: We will be open about the core challenges

Then we are going to focus attention on the understanding we will apply when we write and when we read reports

• We are going to identify what we need to do for reports
• We identify common ways our intuitions can mislead us

Null hypothesis significance testing

Our learning questions are this:

What is NHST and where does it come from?

We are going to learn about hypothesis testing and significance from two different perspectives:

1. The perspective of an individual researcher, asking what they can learn from the evidence from their study
2. The perspective of someone looking at errors that we (in general) could make as we make decisions based on the evidence from multiple repeated studies

NHST is everywhere now but this is recent and controversial

• Through the middle of the 20th century, we completely changed how we did research (Gigerenzer, 1993)

  • We stopped studying individual subjects (almost) and learned to focus on group averages
  • We started doing and reporting null hypothesis tests (using ANOVA or t-test methods)

• NHST is now everywhere (Cumming et al., 2007) but its adoption is controversial (Cassidy et al., 2019) and its utility is disputed (Gigerenzer, 1993; Meehl, 1990)

History, philosophy and psychology

We can get a clearer view if we step back a bit

• The history of ideas

  • NHST embodies a hybrid logic: what motivated the key ideas and what happened next

• Philosophy of science

  • We will reflect on ideas about how we grow knowledge or effective decision-making

• The psychology of us scientists

  • We will look at what people thought researchers wanted or needed to do to make sense of evidence

We begin with Ronald Fisher

Fisher significance test

1. Set up a statistical null hypothesis
2. Calculate a test statistic, given your sample data
3. Report the exact probabibility value \(p\) for this statistic
4. Think about the result: if \(p < .05\) then (Fisher, 1955)

• Either something rare has occurred or null hypothesis is not true

Significance testing: let’s break this into questions

What do you want and need?

Fisher (1990/1991) looked at what researchers want or need

This class might look like it is a statistics class but actually it is about thinking clearly about evidence

• To Fisher (1990/1991), the real issues are to do with the logic any thinking person can use when working out what to do with evidence
• History shows us that what we do is learn from experience
• His methods help us to do this rigorously

Inductive reasoning

You may remember: previously the difference between induction and deduction

1. Induction \(\rightarrow\) From observations to fundamental and general insights, regularities or principles
2. Deduction \(\rightarrow\) To observations or tests, from fundamental and general insights, regularities or principles

Inductive reasoning is a process

Fisher (1990/1991)’s viewpoint, you are an experienced researcher trying to make sense of your evidence:

• Experimental observations are only experience carefully planned in advance
• systematically related to the body of knowledge already acquired
• deliberately observed, with results accurately recorded
• With numbers of trials and quality of methods allowing more or less sensitivity

Inductive reasoning is a process

Fisher (1990/1991)’s viewpoint, you are an experienced researcher trying to make sense of your evidence:

• You know your results will vary — fluctuate — by chance
• so that the things you observe could just be
• a more or less rare random pattern of events or occurrences

So where do we get the rigour from?

• It is possible to argue inductively, given our data

  • from consequences to causes
  • from observations to hypotheses
  • from a sample to the population
  • from the particular to the general

• Any inference from the particular to the general

  • must be associated with some uncertainty
  • but such inference can be absolutely rigorous

The key is the comparison to randomness

The key is the comparison to randomness

There is one way to do this:

1. If you are doing an experiment or making observations for a study, you are doing the work because:

• You think

  • because of previous work or experience or existing knowledge

• that the manipulation of conditions will have some impact

• or that your measurements will be correlated somehow

The key is the comparison to randomness

There is one way to do this:

2. But you have to accept that there are lots of different ways that there might be an impact or an association

• That openness about what the effect or the association might be

  • is \(r\) big or small, positive or negative?

• This openness is vague and it lacks precision

The key is the comparison to randomness

3. What we can be precise about is nothing

• If we propose the theory that the impact of the intervention is nothing or that there is no correlation between variables

• For this kind of proposal we are being absolutely precise about our hypothesis:

  • that the intervention has no impact
  • that the conditions make no difference
  • that there is no correlation between the variables

The key is the comparison to randomness

4. This hypothesis is the null hypothesis and through this null hypothesis, we get to make the comparison to randomness:

• Fisher showed that if we assume the null hypothesis is true
• then we can calculate a probability distribution
• that precisely describes the way in which test results could vary, purely at random
• while allowing for however large or small our sample was

The key is the comparison to randomness

Where we get the rigour from: significance is a measure of disbelief

5. Assuming the null hypothesis is true, we can:

• first calculate or map out what results would look like

  • if they were generated purely by chance
  • given an experimental process like our study process

• then compare the data we observed in our study with this random data pattern

Where we get the rigour from: significance is a measure of disbelief

6. This comparison will make us feel things (Fisher, 1990/1991’s method is about the psychology of researcher thinking):

• we will feel confidence or we will feel disbelief

  • about whether the data we observed could be due to random chance

• and so we will feel confidence or we will feel disbelief

  • about whether the null hypothesis is true

Where we get the rigour from: the significance level

7. Fisher (1990/1991) comments that it is usual and convenient for experimenters to take 5% as a standard level of significance: what does this mean?

• Most of the time, we might see results that look very much like the results that we would see

  • If we assume the null hypothesis is true
  • And if we calculate, given that null hypothesis
  • a distribution describing all the different ways our results could turn out at random

Where we get the rigour from: the significance level

• But some times, the results we observe are so different from what we would see

  • If we assume the null hypothesis is true
  • And if we calculate, given that null hypothesis, a distribution describing all the different ways that our results could turn out at random

• That it would be hard to believe, actually, that the null hypothesis is true

The significance test

8. What we see here is Fisher constructing a method for inductive reasoning and so a way for you the scientist to think clearly about your evidence

• We define a precise hypothesis, the null hypothesis

• We observe our data and calculate our results

• We compare these observed results to randomness:

  • the distribution of results we would get
  • by chance, supposing the null hypothesis is true

The significance test

• If the probability that our observed results look like the null-is-true random-distributed-results is small

  • 5% probability or less

• Then we have evidence for disbelieving that the null hypothesis is true

The significance test

Demonstration: significance testing

We have outlined the logic of the significance test in words

Significance test for correlation

We see how significance tests work by simulating a study

1. Your history of reading (print exposure or reading experience) could be linked to your development of knowledge about words (vocabulary knowledge) (Stanovich et al., 1995; Vermeiren et al., 2023)

• maybe because reading more means you are more exposed to different words and ideas about things in the world
• and neural networks adapt in response to experience

Significance test for correlation

2. For this demonstration, we are going to work with the potential correlation (the association) between two measures of individual differences

• variation in scores on a test of vocabulary
• variation in scores on a test of reading experience

You test a person and record their score on a variable

Let’s imagine we test a person (you) on a psychological attribute like your vocabulary knowledge

You test two people and record their scores

Now imagine we test two people (you and your neighbour) using the same vocabulary test

You test a sample of people

Now imagine we test a sample of 30 people

Test a sample of people and record their scores on two variables

For the same people, we record scores on (1.) vocabulary and (2.) reading experience

What is the association between vocabulary and reading experience?

• Each point represents a person’s reading experience compared to vocabulary

  • High experience scores are near the right
  • High vocabulary scores are near the top

• Is high experience associated with high vocabulary?

What is the correlation between vocabulary and reading experience?

The correlation \(r\) between these variables is:

```{r}
library(correlation)
cor_test(data, "experience", "vocabulary")
```

Parameter1 | Parameter2 |    r
------------------------------
experience | vocabulary | 0.54

Significance test for correlation

1. Specify the null hypothesis that:

• The correlation between these two variables is zero
• \(r=0\) for the correlation between vocabulary and reading experience

2. Calculate a statistical test of the null hypothesis: t-test

   \[ t = \frac{r\sqrt{N-2}}{\sqrt{1-r^2}} \]

Significance test for correlation

3. Calculate the probability value for this t-test statistic

• When earlier we talked about comparing our observed results to randomness:

  • the distribution of results we would get
  • by chance, supposing the null hypothesis is true

• This is what we meant

Significance test for correlation

\[ t = \frac{r\sqrt{N-2}}{\sqrt{1-r^2}} \]

• Compare this t-test statistic

  • calculated given the \(r\) and \(N\) for our data

• With a probability distribution describing all the different ways:

  • that results could turn out at random

    • given the same situation
    • assuming that the null hypothesis is true

Significance test for correlation

\[ t = \frac{r\sqrt{N-2}}{\sqrt{1-r^2}} \]

• Compare this t-test statistic

  • calculated given the \(r\) and \(N\) for our data

• With a probability distribution describing all the different ways:

  • that t-statistics could vary

    • given the same situation
    • assuming that the null hypothesis is true

Significance test for correlation

Suppose the null hypothesis is true

Imagine we run the study over and over again, drawing a different sample of people each time

Figure 10: Six studies, each with its own sample of 30 people

Suppose the null hypothesis is true

Figure 11: Six simulated studies: for each, the null hypothesis is true: \(r = 0)\)

Suppose the null hypothesis is true

For each study, we calculate the correlation \(r\)

Figure 12: The six studies’ r values: the null hypothesis is true; the numbers inside the dots are the study number

Suppose the null hypothesis is true

For each study, we calculate the t-test on the correlation, and record \(t\)

Figure 13: The six studies’ t values; the numbers inside the dots are the study number

Counting to get probability

• We can compare the results for our observed sample data with a distribution showing how results vary at random

  • for the same situation
  • given the null hypothesis is true

• by simulating many studies in which we collect data from participants on vocabulary and reading experience

Counting to get probability

• If every time we did this, we calculate:

  • the correlation between these variables
  • the \(t\) statistic for this correlation

• and we record all the t-statistic values

• eventually, we would get a distribution of \(t\) statistics

Counting to get probability

• This distribution of \(t\) statistics would describe for us how results would vary at random supposing the null hypothesis is true

• We can then count the number of times different values of \(t\) occur

• This count shows the probability of different values of \(t\) occurring as a relative frequency

  • Sometimes, a \(t\) will occur very often
  • Very large values of \(t\) will occur less often

Suppose the null hypothesis is true

We simulate a 1000 studies

Figure 14: A distribution of different \(t\) values from 1000 simulated studies. In all simulated studies, the null hypothesis is true

Now compare the \(t\) statistic for our observed data to the null hypothesis distribution of \(t\) statistics

Suppose the null hypothesis is true

We simulate a 1000 studies

Figure 15: The same 1000 \(t\) values, with lines marking the critical \(t\) (where \(p = .05\), in grey) and our observed study’s \(t\) (marked in red)

Significance testing

We compare the \(t\) statistic for our observed data to the null hypothesis distribution of \(t\) statistics

• If few values of \(t\) in the null hypothesis distribution

  • are as large as the \(t\) statistic for our observed data (or larger)

• then that precisely is how we calculate how small the probability for the \(t\) statistic for our data is

Significance testing

In the inductive reasoning that this test allows

• If the probability is small (\(p < .05\)) we can say\(\rightarrow\)
• Then either an improbable event has occurred or the null hypothesis is not verified

The significance test

Hypothesis testing

We switch to a different philosophy and a different view of the psychology of researchers (Hubbard, 2004): we look at the ideas of Jerzy Neyman and Egon Pearson

Jerzy Neyman and Egon Pearson

Inductive behaviours

• Jerzy Neyman and Egon Pearson did not think we can draw conclusions about the truth or falsity of our hypotheses
• We can make decisions, given our data, and behave accordingly

Inductive behaviours

• This is a method for adjusting behaviours, given the data

• Neyman and Pearson’s recommendations on decision-making rules

• are based on thinking about the psychology of humans (and dogs)

  • Just like Pavlov’s dogs, we can adjust our actions “to avoid undesirable consequences” (Hubbard, 2004; Neyman, 1962)

Null versus alternate hypotheses

• Neyman and Pearson did not think that the “rational mind” can reject a null hypothesis without a plausible alternative

1. So the researcher proposes:

• The null hypothesis: \(H_0\)
• The alternative hypothesis: \(H_A\)

2. Hypothesis testing is testing the null against the alternative

Null versus alternate hypotheses

In our example about the correlation between vocabulary and reading experience

• The null hypothesis \(H_0\) can correspond to no correlation
• The alternative hypothesis \(H_A\) can correspond to some correlation

A balance of errors

In hypothesis testing, we balance the risks of making different kinds of errors:

1. False rejection of the null hypothesis, if null hypothesis is correct

   • Type 1 errors, also known as false positives

2. False acceptance of the null hypothesis, if alternative hypothesis is correct

   • Type 2 errors, also known as false negatives

A balance of error rates

In hypothesis testing, we balance the risks of making different kinds of errors:

1. We can assume: some chance that we will make a Type 1 error

• The probability of making a Type 1 error is \(\alpha\) (“alpha”)

2. We can assume: some chance that we will make a Type 2 error

• The probability of making a Type 2 error is \(\beta\) (“beta”)

Change your perspective

We change perspectives here:

• You are not now the researcher, thinking about the data from your study, and wondering whether it inspires you to disbelieve your null hypothesis

Change your perspective

We change perspectives here:

• You are the owner of a psychology factory or the queen of psychologists or a scientist analyzing the quality of evidence of a field of research

  • And you are looking at a stream or a production line or a field of like studies

    • one study completed after the other
    • one set of results following the other

Change your perspective

• What you are worried about is quality control over this long run of studies (Gigerenzer, 1993; Hubbard, 2004)

  • You know that mistakes will happen, due to chance
  • But you want to get information you can use for your decision-making as efficiently as possible

• What you want to do is balance the risk of Type 1 and Type 2 errors

Change your perspective

• The balance can vary in different situations because the costs of different errors can vary in different situations

• Most of the time, on balance, you are more worried about false positives (Type 1 errors) than false negatives (Type 2 errors)

  • And because you are more worried about false positives, you fix the Type 1 error rate you are prepared to tolerate to a low level:

    • \(\alpha = .05\) or equivalently \(\alpha = 5\%\)

The meaning of \(\alpha\) (“alpha”)

• What we say we are doing here is setting the significance levels

• If we set the significance level to \(\alpha = 5\%\) this means (Gigerenzer, 1993):

  • If the null hypothesis is correct
  • and the experiment is repeated many times
  • then the experimenter risks wrongly rejecting the null in \(5\%\) of cases

Key differences between Fisher-style and Neyman-Pearson-style

What are we talking about when we talk about significance level?

• For Fisher, the exact level of significance is a property of the data:

  • How different are the observed data compared to a random pattern?

• For Neyman and Pearson, the \(\alpha\) (“alpha”) significance level is a property of the test:

  • What error rate are you willing to tolerate?

Key differences between Fisher-style and Neyman-Pearson-style

• If \(H_0\) is accepted and \(H_A\) is rejected

  • though the alternative hypothesis is true
  • this is a Type 2 error

• We can balance the risk of Type 2 errors (the \(\beta\) error rate) by planning for the power of our test

Hypothesis testing

Seeing the Neyman-Pearson approach at work

To help us to build understanding, we work through demonstrations based on simulations of Psychology studies of correlation

Seeing Neyman-Pearson at work

• We ask what does it look like if:

  1. Scenario One: The null hypothesis is true: \(r = 0\)

     • Correlation between vocabulary and experience is zero

  2. Scenario Two: The alternative hypothesis is true: e.g. \(r = .5\)

     • Correlation between vocabulary and experience is .5

• What are our chances of making a Type 1 (false positive) error in scenario 1?

• What are our chances of making a Type 2 (false negative) error in scenario 2?

Scenario One: \(H_0\) is true

We simulate a series of studies. In each study, we

• Test a sample of 30 people

• Record their scores on two variables

  • Measure of reading experience
  • Measure of vocabulary

Scenario One: \(H_0\) is true

We simulate a series of studies

• In all these studies, the null hypothesis is true

  • The correlation between vocabulary and experience for all these studies is zero

• We set the \(\alpha\) (Type 1 error rate) to \(\alpha = .05\)

Scenario One: \(H_0\) is true

• What will we see?

  • For each study, what is the correlation \(r\) between vocabulary and experience? We calculate \(r\)

• We calculate the t-test for each study correlation

  • For each study, is \(t\) significant at the \(\alpha = .05\) significance level?

\(H_0\) is true but do we see correlations appearing by chance?

Figure 16: Scatterplots of the data from six simulated studies: are high experience scores associated with high vocabulary scores? In all studies, the null hypothesis is true (r = 0). (Note: we saw these data in the earlier discussion of Fisher’s approach.)

\(H_0\) is true but do we see correlations appearing by chance?

Figure 17: The six studies’ r values, coloured by whether the correlation is significant at \(\alpha = .05\). We saw these data in the earlier discussion of Fisher’s approach.

\(H_0\) is true: from \(r\) to the test statistic \(t\)

Figure 18: The same six studies’ \(t\) statistics. Critical values for significance in relation to \(t\) values, at \(\alpha = .05\).

\(H_0\) is true: Do we see significant correlations appearing by chance?

Figure 19: Distribution of \(r\) values from 1000 simulated studies. In each study, data are simulated given the assumption that the null hypothesis is true (r = 0). Points are coloured by whether the \(t\) test statistic for the \(r\) is significant (or not) at \(\alpha = .05\)

\(H_0\) is true: Do we see significant correlations appearing by chance?

Figure 20: The \(t\) statistics for the same 1000 studies. Critical values are defined in relation to \(t\) values: \(t\) values beyond the dashed lines are significant at \(\alpha = .05\).

\(H_0\) is true but do we see correlations appearing by chance?

• In all 1000 studies, data are simulated allowing experience and vocabulary scores to vary at random

  • But assuming the null hypothesis is true so \(r = 0\)

• We see that if we do a t-test on each correlation for each study

  • some \(t\) statistics will be significant at the \(\alpha = .05\) significance level
  • purely as a result of random variation

Scenario Two: \(H_1\) is true

We again simulate a series of studies. In each study, we

• Test a sample of 30 people

• Record their scores on two variables

  • Measure of reading experience
  • Measure of vocabulary

Scenario Two: \(H_1\) is true

We simulate a series of studies

• In all these studies, the alternative hypothesis is true

  • The correlation between vocabulary and experience for all these studies is \(.5\)

• We set the \(\beta\) (Type 2 error rate) to \(\beta = .20\)

Scenario Two: \(H_1\) is true

What will we see?

• For each study, what is the correlation \(r\) between vocabulary and experience? We calculate \(r\)

  • We calculate the t-test for each study correlation

• For each study, is \(t\) significant at the \(.05\) level?

Scenario Two: \(H_1\) is true, do we see the correlations?

Figure 22: Six studies, each simulated assuming that the alternative hypothesis is true (r = .5). (We have not seen these data before.)

Scenario Two: \(H_1\) is true, do we see the correlations?

Figure 23: The \(r\) values for the same six studies. The point are coloured by whether the correlation is significant at the \(\alpha = .05\) significance level

Scenario Two: \(H_1\) is true, from \(r\) to the test statistic \(t\)

Figure 24: The same six studies’ \(t\) statistics. Critical values for significance are defined on the \(t\) scale, at \(\alpha = .05\); the dotted line marks the \(t\) corresponding to the true (population) correlation, \(r = .5\).

Scenario Two: \(H_1\) is true, do we see significant correlations?

Figure 25: Distribution of \(r\) values from 1000 simulated studies. In each study, data are simulated given the assumption that the alternative hypothesis is true (\(r = .5\)). Points are coloured by whether the \(t\) test statistic for the \(r\) is significant (or not) at \(\alpha=.05\) and \(\beta = .20\)

Scenario Two: \(H_1\) is true, do we see significant correlations?

Figure 26: The \(t\) statistics for the same 1000 studies. Critical values are defined in relation to \(t\) values: \(t\) values beyond the dashed lines are significant at \(\alpha = .05\).

Scenario Two: \(H_1\) is true, do we see significant correlations?

• In all 1000 studies, data are simulated allowing experience and vocabulary scores to vary at random

  • But assuming the alternative hypothesis is true so \(r = 0.5\)

• We see that if we do a t-test on each correlation for each study

  • most \(t\) statistics will be significant at the \(\alpha = .05\) significance level
  • but some \(t\) statistics will not be significant purely as a result of random variation

Thinking about power

• With larger samples, we get fewer false positives

• We can prove the point by, again, simulating correlation studies

  • If we assume the null hypothesis is true so \(r = 0\)

1. \(r\) estimates of correlation more variable when samples are smaller

2. and associated \(t\) statistics for some of these \(r\) estimates will be significant at the \(\alpha = .05\) significance level

   • purely as a result of random variation

Thinking about power

• With larger samples, we get fewer false negatives

• We can prove the point by, again, simulating correlation studies

  • If we assume the alternative hypothesis is true so \(r = 0.5\)

1. \(r\) estimates of correlation more variable when samples are smaller

2. and associated \(t\) statistics for some of these \(r\) estimates will not be significant at the \(\alpha = .05\) significance level

   • purely as a result of random variation

Hypothesis testing

The NHST hybrid in textbooks and research reports

• In textbooks, we often see presentations of Null Hypothesis Significance Testing that do not explain where the ideas come from or what the motivations are

  • NHST presentations often mash together Fisher’s ideas and the ideas of Neyman and Pearson

• We have seen that the ideas align with different perspectives on how we should respond to evidence

  • How we think about what our evidence shows
  • How we make decisions based on our evidence

The NHST ritual

Gerd Gigerenzer identifies Null Hypothesis Significance Testing (NHST) as a hybrid logic (Gigerenzer, 1993) that we apply in ritualistic behaviours (Gigerenzer, 2018):

1. Set up a null hypothesis of “no mean difference” or “zero correlation”
2. Use 5% as a convention for rejecting the null hypothesis
3. Report the test result as \(p <.05\), \(p < .01\), or \(p < .001\)

The impact on our work: reporting

• You will want identify:

  • What is your null hypothesis?
  • Do you have an alternative hypothesis: what is it?
  • What risk levels you are prepared to tolerate: \(\alpha = .05, \beta = .20\)

• You calculate the correlation \(r\) and the test statistic for the correlation \(t\)

• You indicate the sample size (in statistics, we use \(df\), degrees of freedom)

• You report these statistics with the \(p\) value

The impact on our work: thinking