1. Experimental Research Strategy & Between-Participants Designs

Mark Hurlstone

Lectures

Watch Lecture 1 Part 1 here

Watch Lecture 1 Part 2 here

Watch Lecture 1 Part 3 here

Download the Lecture 1 slides here, and here for a larger version.

Watch Lecture 2 Part 1 here

Watch Lecture 2 Part 2 here

Watch Lecture 2 Part 3 here

Download the Lecture 2 slides here, and here for a larger version.

This week, you will explore how experiments establish cause-and-effect relationships through manipulation, measurement, and control, and why correlation alone cannot demonstrate causation. You will also examine between-participants designs, including their advantages and limitations, the challenges posed by individual differences and confounding variables, and strategies for creating equivalent groups. The goal is to understand how to design experiments that support valid conclusions. Before attending the practical class, please review the Week 1 lecture materials and recordings above. There are some optional multiple-choice questions below to help you consolidate your learning and it is recommended to complete these once you have viewed the lecture recordings.

Multiple-Choice Questions for Lecture 1: Experimental Research Strategy

The following questions are designed to help you consolidate your understanding of the material covered in this week’s lectures. Try to answer each question before revealing the answer and feedback.

Question 1

How do studies using the experimental research strategy differ from other types of research?

  1. Only experiments can demonstrate a cause-and-effect relationship between variables.
  2. Only experiments involve comparing two or more groups of scores.
  3. Only experiments can demonstrate that relationships exist between variables and provide a description of the relationship.
  4. Only experiments can demonstrate a bidirectional relationship between variables.

Answer: 1. Only experiments can demonstrate a cause-and-effect relationship between variables.

The key advantage of the experimental research strategy is that manipulation and control allow researchers to establish cause-and-effect relationships.

Question 2

A researcher is interested in studying how indoor lighting can influence people’s moods during the winter. A sample of 100 households is selected. Fifty of the homes are randomly assigned to the bright light condition where the researcher replaces all the light bulbs with 100-watt bulbs. In the other 50 houses, all the lights are changed to 60-watt bulbs. After two months, the researcher measures the level of depression for the people living in the houses.

In this example, how many dependent variables are there?

  1. 100
  2. 50
  3. 2
  4. 1

Answer: 4. 1

There is one dependent variable: level of depression. The number of participants or experimental conditions does not determine the number of dependent variables.

Question 3

Research indicates that people who suffer from depression also tend to experience insomnia. However, it is unclear whether the depression causes insomnia or the lack of sleep causes depression. What problem is demonstrated by this example?

  1. The directionality problem
  2. The third-variable problem
  3. The extraneous variable problem
  4. The manipulation check problem

Answer: 1. The directionality problem

The relationship does not tell us which variable causes the other: depression could contribute to insomnia, or insomnia could contribute to depression.

Question 4

In an experiment, what is the purpose for manipulating the independent variable?

  1. It helps establish the direction of the relationship by showing that the dependent variable changes when you manipulate the independent variable.
  2. It helps eliminate the third-variable problem because you decide when to manipulate rather than waiting for the variable to change.
  3. It helps establish the direction of the relationship and it helps eliminate the third-variable problem.
  4. Manipulation does not establish the direction of the relationship or eliminate the third-variable problem.

Answer: 3. It helps establish the direction of the relationship and it helps eliminate the third-variable problem.

By actively manipulating the independent variable, the researcher establishes which variable changes first and avoids relying on naturally occurring differences that might be associated with other variables.

Question 5

To establish an unambiguous relationship between two variables, it is necessary to eliminate the possible influence of which of the following variables?

  1. Extraneous variables
  2. Confounding variables
  3. Independent variables
  4. Dependent variables

Answer: 2. Confounding variables

A confounding variable provides an alternative explanation for an observed relationship between the independent and dependent variables. Experiments therefore need to prevent extraneous variables from becoming confounding variables.

Question 6

Which of the following characteristics are necessary for an extraneous variable to become a confounding variable?

  1. It must change systematically from one participant to the next.
  2. It must change systematically when the independent variable is changed.
  3. It must have no systematic relationship with the dependent variable.
  4. It must have no systematic relationship to either the independent or the dependent variables.

Answer: 2. It must change systematically when the independent variable is changed.

An extraneous variable becomes a potential confound when it varies systematically with the independent variable, meaning that differences in the dependent variable could potentially be attributed to the extraneous variable rather than the manipulation.

Question 7

In an experiment comparing two treatments, the researcher assigns participants to treatment conditions so that each condition has fifteen 7-year-old children and ten 8-year-old children. For this study, what method is being used to control participant age?

  1. Randomisation
  2. Matching
  3. Holding constant
  4. Limiting the range

Answer: 2. Matching

The researcher is ensuring that the two conditions contain the same numbers of children of each age. The groups are therefore matched with respect to participant age.

Question 8

Holding a variable constant is a technique for removing one threat to ___________, but it can limit the ___________ of an experiment.

  1. internal validity, external validity
  2. external validity, internal validity
  3. internal validity, reliability
  4. external validity, reliability

Answer: 1. internal validity, external validity

Holding an extraneous variable constant can improve internal validity by preventing it from becoming a confound. However, restricting the circumstances or participants included in an experiment can reduce its external validity.

Question 9

Which of the following is the primary goal for randomly assigning participants to treatment conditions in an experiment?

  1. Increase the ability to generalize the results.
  2. Avoid selection bias.
  3. Ensure that the individuals in the sample are representative of the individuals in the population.
  4. Minimise the likelihood that a participant variable (such as age or gender) becomes a confounding variable.

Answer: 4. Minimise the likelihood that a participant variable (such as age or gender) becomes a confounding variable.

Random assignment is intended to disrupt systematic relationships between participant characteristics and the experimental conditions, thereby reducing the likelihood that participant variables become confounds.

Question 10

What is the purpose for using a control condition in an experiment?

  1. It provides a baseline that can be used to evaluate the size of the treatment effect.
  2. It minimizes the threat of a confounding variable.
  3. It is necessary to ensure the internal validity of the study.
  4. It is necessary to ensure the external validity of the study.

Answer: 1. It provides a baseline that can be used to evaluate the size of the treatment effect.

A control condition provides a comparison or baseline against which the effect observed in the treatment condition can be evaluated.

Question 11

An experiment includes a treatment condition, a no-treatment control, and a placebo control. Which two conditions should be compared to determine the size of the effect that is actually caused by the treatment?

  1. Placebo vs. treatment
  2. Placebo vs. no treatment
  3. Treatment vs. no treatment
  4. You only need to look at the scores in the placebo control condition.

Answer: 1. Placebo vs. treatment

Comparing the treatment condition with the placebo condition allows the researcher to estimate the effect attributable to the treatment itself while accounting for effects associated with receiving or expecting a treatment.

Question 12

A researcher exposes people to a stressful situation (such as public speaking) to examine the effect of stress on depressed mood. Why would the researcher also include a measure of stress?

  1. It is a measure of the dependent variable.
  2. It is a measure of extraneous variables.
  3. It is a control for confounding variables.
  4. It is a manipulation check.

Answer: 4. It is a manipulation check.

The additional measure allows the researcher to determine whether the experimental manipulation successfully affected the intended construct—in this case, whether the stressful situation actually increased stress.

Multiple-Choice Questions for Lecture 2: Between-Participants Designs

The following questions are designed to help you consolidate your understanding of the material covered in this week’s lectures. Try to answer each question before revealing the answer and feedback.

Question 1

Which statement best characterises a between-participants experimental design?

  1. Participants are randomly selected from two different populations.
  2. Each participant is assigned to one condition of the experiment.
  3. Each participant is assigned to every condition of the experiment.
  4. Participants with different characteristics are assigned to the different conditions of the experiment.

Answer: 2. Each participant is assigned to one condition of the experiment.

In a between-participants design, each participant contributes data to only one experimental condition. Different participants are therefore used in the different conditions.

Question 2

Which of the following accurately describes the scores in a between-participants experiment?

  1. Only one score is obtained for each participant.
  2. At least two scores are obtained for each participant.
  3. One score is obtained for each treatment condition for each participant.
  4. Each score represents multiple participants.

Answer: 1. Only one score is obtained for each participant.

Because each participant takes part in only one treatment condition, each score is obtained from a different participant.

Question 3

If a between-participants experiment produces 50 scores in treatment 1 and 50 scores in treatment 2, then how many participants were in the entire experiment?

  1. 50 participants
  2. 200 participants
  3. 25 participants
  4. 100 participants

Answer: 4. 100 participants

Each score comes from a different participant in a between-participants design. Therefore, 50 scores in each of two conditions require 100 participants in total.

Question 4

Which of the following does not guarantee that a specific participant variable will not become a confounding variable?

  1. Matching the variable across treatments
  2. Randomising the variable across treatments
  3. Holding the variable constant
  4. All of the above options guarantee that the variable does not become a confounding variable.

Answer: 2. Randomising the variable across treatments

Random assignment does not guarantee that groups will be equivalent on a particular participant variable. Especially with small samples, chance can produce differences between groups. Matching or holding a specific variable constant provides more direct control over that variable.

Question 5

In an experiment comparing two treatments, the researcher assigns participants to treatment conditions so that the average participant age in each is approximately the same. For this study, what method is being used to control participant age?

  1. Randomisation
  2. Matching
  3. Holding constant
  4. Limiting the range

Answer: 2. Matching

The groups are being constructed so that they are equivalent with respect to average participant age. This is an example of matching.

Question 6

Which of the following is a limitation of using matching rather than random assignment to form groups in a between-participants design?

  1. Matching requires another measurement procedure.
  2. Matching reduces error due to participant differences.
  3. Matching is easier than randomisation.
  4. Matching eliminates any systematic relationship between participant characteristics and the treatment conditions.

Answer: 1. Matching requires another measurement procedure.

To match participants or groups on a particular characteristic, the researcher must first measure that characteristic. Matching can therefore require additional time and measurement compared with simple random assignment.

Question 7

How does holding a variable constant prevent the variable from becoming a confound?

  1. It eliminates the possibility that the variable will be substantially different from one group to another.
  2. It reduces error.
  3. It ensures a nonbiased sample.
  4. It increases the differences between the groups.

Answer: 1. It eliminates the possibility that the variable will be substantially different from one group to another.

If a variable is held constant, it cannot vary systematically between the experimental conditions and therefore cannot provide an alternative explanation for differences between those conditions.

Question 8

Which of the following maximises the likelihood of a successful research result?

  1. Decrease the differences between treatments and increase the variance within treatments.
  2. Increase the differences between treatments and increase the variance within treatments.
  3. Increase the differences between treatments and decrease the variance within treatments.
  4. Decrease the differences between treatments and decrease the variance within treatments.

Answer: 3. Increase the differences between treatments and decrease the variance within treatments.

Treatment effects are easiest to detect when the differences between treatment conditions are large relative to the variability of scores within those conditions.

Question 9

Which of the following is an option for limiting the variance within treatment conditions?

  1. Hold a participant variable constant.
  2. Randomise participant variables across treatments.
  3. Match participant variables across treatments.
  4. All of the above are options for limiting variance within treatments.

Answer: 1. Hold a participant variable constant.

Holding a participant variable constant reduces individual differences within each treatment condition. Randomisation and matching can help balance participant characteristics between conditions, but they do not necessarily reduce variability within conditions.

Question 10

Which of the following is a potential problem with holding a participant variable constant?

  1. It threatens the reliability of the study.
  2. It threatens the external validity of the study.
  3. It lowers the likelihood of obtaining a significant difference between treatments.
  4. None of the above is a potential problem.

Answer: 2. It threatens the external validity of the study.

Holding a participant variable constant restricts the range of individuals represented in the study. This can limit the extent to which the findings generalise to people with other characteristics.

Question 11

Which of the following accurately defines why differential attrition is a threat to the internal validity of a between-participants study?

  1. Groups are not created equally.
  2. Groups are not treated equally.
  3. Groups may not be composed of equivalent individuals.
  4. All of the above.

Answer: 3. Groups may not be composed of equivalent individuals.

Differential attrition occurs when participant dropout differs across conditions. Even if the groups were initially equivalent, differential dropout can result in groups that differ systematically in their composition.

Question 12

Which of the following accurately defines compensatory equalization?

  1. One group demands the same benefits received by another group.
  2. One group works extra hard to make up for not receiving the benefits received by another group.
  3. One group stops trying because it is not receiving the benefits received by another group.
  4. Elements of the treatment in one group have spread to another group.

Answer: 1. One group demands the same benefits received by another group.

Compensatory equalization occurs when differences in the treatment received by different groups lead to efforts to provide one group with benefits comparable to those received by another.

Question 13

Which of the following accurately defines compensatory rivalry?

  1. One group demands the same benefits received by another group.
  2. One group works extra hard to make up for not receiving the benefits received by another group.
  3. One group stops trying because it is not receiving the benefits received by another group.
  4. Elements of the treatment in one group have spread to another group.

Answer: 2. One group works extra hard to make up for not receiving the benefits received by another group.

Compensatory rivalry occurs when participants who know they are not receiving a desirable treatment respond by increasing their effort or performance.

Question 14

Which of the following accurately defines diffusion?

  1. One group demands the same benefits received by another group.
  2. One group works extra hard to make up for not receiving the benefits received by another group.
  3. One group stops trying because it is not receiving the benefits received by another group.
  4. Elements of the treatment in one group have spread to another group.

Answer: 4. Elements of the treatment in one group have spread to another group.

Diffusion occurs when aspects of a treatment intended for one condition become available to participants in another condition, reducing the intended difference between the groups.

Practical

This week’s lab

In this week’s lab, we will revisit some of the R skills introduced in Year 1. Using data from a hypothetical study comparing two robot lab assistants, we will practise inspecting and organising data, calculating descriptive statistics, visualising group differences, and conducting an independent-samples t-test. We will also revisit how we measure variability in scores, including the relationship between the sum of squares, variance, and standard deviation. These ideas will support our work on Analysis of Variance (ANOVA) in subsequent weeks.

By the end of the session, you should be able to:

  • Import a data set and inspect, sort and transform its variables.
  • Calculate and interpret descriptive statistics for the full sample and for separate groups.
  • Explain the relationship between variance and standard deviation.
  • Produce a box plot and interpret an independent-samples t-test and Cohen’s d.
  • Report the findings in a short results paragraph.

Work through the examples in your R script, paying attention to the code comments and the resulting output. Attempt each exercise before opening the answer. If you get stuck, ask one of the GTAs for help.

Getting started

To log in to the R server, first make sure that you have the VPN switched on, or you will need to be connected to the university network (Eduroam). To set up the VPN, follow ISS instructions here or connecting to Eduroam here.

When you are connected, navigate to https://ps-rstudio.lancaster.ac.uk/, where you will be shown a login screen that looks like the below. Click the option that says “Sign in with SAML”.

This will take you through to the University login screen, where you should enter your username (e.g. ivorym) and then your university password. This will then redirect you to the R server where you can start using RStudio!

Note

If you have already logged in through the university login, perhaps to get to the portal, then you may not see the username/password screen. When you click login, you will be redirected straight to RStudio. This is because the server shares the login information securely across the university.

  1. In RStudio’s Files pane, click Home to open your home directory on the server. Select New Folder and create a folder named psyc5141_labs. Use this folder to organise your work for the module.
  2. Open psyc5141_labs and create a folder inside it named psyc5141_lab_1. This will contain the data, project and script for today’s session.
  3. Download the Week 1 data file and upload it to psyc5141_lab_1.
  4. Create an RStudio project using File → New Project → Existing Directory. Select the psyc5141_lab_1 folder inside psyc5141_labs, then click Create Project.
  5. Open a new R script and save it in the project folder as week1_analysis.R.

A project keeps your work associated with this folder and sets it as the working directory, allowing R to find your data files. The working directory is also the default location R uses to save files. You can check it by running getwd(); it should end in psyc5141_labs/psyc5141_lab_1. With the project open, you do not need a separate setwd() command.

When returning to the exercise, open the project before running your script. In the server’s Files pane, navigate to psyc5141_labs/psyc5141_lab_1 and open psyc5141_lab_1.Rproj. For subsequent labs, create a new weekly folder inside psyc5141_labs.

Let’s load the packages we will use:

library(tidyverse) # Data manipulation and plotting
library(rstatix)   # Summary statistics and effect sizes
library(ggpubr)    # Box plots

Next, import the data:

# Import the CSV from your project folder
lab1_data <- read_csv("week1_robo_lab.csv")

The assignment operator <- stores the imported data in an object called lab1_data. We will use this name throughout the session.

Meet the robot study

A team of researchers wants to know whether the design of a robot lab assistant affects students’ learning of statistics and their attitudes towards the assistant. They develop two prototypes, Alpha and Beta, and randomly assign each student to work with one of them. Robot Beta has a more human-like appearance than Robot Alpha.

Robot A(lpha)

Robot B(eta)

This is a single-factor between-participants design: each participant experiences only one condition. The study has one factor (another term for an independent variable), robot assignment, with two levels, Alpha and Beta.

To evaluate which robot would be more suitable as a lab assistant, the researchers needed to assess both how well students learned and how positively they responded to the robot. A robot might be effective at supporting learning without being particularly well liked, or well liked without being especially effective.

The researchers therefore measured two dependent variables, or outcomes. First, students completed a statistics test, scored from 0–100, to assess their learning. Second, students rated how much they liked their robot lab assistant on a scale from 1 (strongly dislike) to 7 (strongly like).

Variable Meaning
ID Participant identifier.
Group Robot assignment: A for Alpha and B for Beta.
Score Statistics test score with a possible range of 0–100.
Likeability Rating of the robot from 1 (strongly dislike) to 7 (strongly like).

The researchers expected that Robot Beta’s more human-like appearance would lead students to respond more positively to it and learn more effectively when working with it. They therefore predicted that students assigned Robot Beta would obtain higher statistics test scores and give higher likeability ratings than students assigned Robot Alpha. These are two separate hypotheses, corresponding to the two dependent variables.

We will retain this study in subsequent weeks, when the researchers introduce a third robot and we extend the analysis to three groups.

NoteKey idea: factors and dependent variables

A factor is an independent variable whose levels define the conditions being compared. A dependent variable is the outcome measured for each participant. Here, Group identifies the levels of our factor, while Score and Likeability contain the two outcomes.

Exploring the data

Inspecting and sorting rows

Let’s begin by looking at the first six rows:

head(lab1_data)
# A tibble: 6 × 4
     ID Group Likeability Score
  <dbl> <chr>       <dbl> <dbl>
1     1 A               3    55
2     2 A               2    72
3     3 A               2    45
4     4 A               3    50
5     5 A               2    45
6     6 A               5    63

The head() function displays six rows by default. To display twelve, use head(lab1_data, n = 12). You can also inspect the full data set in RStudio using View(lab1_data).

To display the ten lowest test scores, we can combine arrange() and head():

lab1_data %>%
  arrange(Score) %>%  # Sort from lowest to highest Score
  head(n = 10)       # Display the first ten rows
# A tibble: 10 × 4
      ID Group Likeability Score
   <dbl> <chr>       <dbl> <dbl>
 1    33 A               2    40
 2    60 A               1    40
 3   165 B               5    40
 4    81 A               3    41
 5    46 A               3    42
 6    52 A               2    42
 7    18 A               4    43
 8    44 A               3    43
 9    95 A               3    43
10    50 A               3    44

The pipe, %>%, passes the result of one step to the next. Here, we start with lab1_data, sort its rows by Score, and display the first ten rows of the sorted data. This displays a sorted version without changing the original object.

TipTry it yourself: the highest scores

Adapt the code to display the ten highest test scores. Are these scores mainly associated with one robot group? Does inspecting only the ten highest scores tell us how the groups compare overall?

Use arrange(desc(Score)) to sort scores in descending order.

lab1_data %>%
  arrange(desc(Score)) %>%
  head(n = 10)
# A tibble: 10 × 4
      ID Group Likeability Score
   <dbl> <chr>       <dbl> <dbl>
 1   175 B               5    88
 2   178 B               5    88
 3   134 B               5    86
 4   183 B               4    85
 5    10 A               1    83
 6   156 B               4    83
 7   145 B               4    82
 8   108 B               3    81
 9   124 B               4    81
10   138 B               5    81

Nine of the ten highest scores belong to Group B (Beta), and one belongs to Group A (Alpha). These rows describe only the highest-scoring participants. To compare the groups overall, we need to examine all their observations, including their average scores and variability.

Creating a variable

So far, we have inspected variables that were already present in the data set. Sometimes we need to create a new variable from an existing one. For example, suppose we wanted to label each participant’s test performance as Low, Medium, or High. We already have their numerical test score in Score; what we want is an additional column containing the corresponding label for each participant.

For this exercise, we will classify scores of 50 or below as Low, scores above 50 and up to 70 as Medium, and scores above 70 as High. A participant who scored 45 would therefore receive the label Low, while a participant who scored 65 would receive Medium. The original numerical score will remain in the data set.

We can do this using mutate(), which adds or changes columns in a data frame. Inside it, case_when() tells R which label to use for each score:

# Update the data frame in R with the new column
lab1_data <- lab1_data %>%
  # Create a new column using these rules
  mutate(Score_Category = case_when(
    Score <= 50 ~ "Low",              # Scores of 50 or below receive Low
    # Scores above 50 and up to 70 receive Medium
    Score > 50 & Score <= 70 ~ "Medium",
    Score > 70 ~ "High"               # Scores above 70 receive High
  ))

Assigning the result back to lab1_data updates the data frame in R, adding the new Score_Category column. It does not change the original CSV file.

R checks these rules for each row, which corresponds to one participant. Each line inside case_when() gives a condition on the left of ~ and the label to assign on the right. For example, Score <= 50 ~ "Low" means “if this participant’s score is less than or equal to 50, enter Low”. The & means that both conditions must be satisfied.

Score_Category is the name we have chosen for the new column. Creating variables in this way is useful when we need to recode responses or derive new measures from information we have already collected.

TipTry it yourself: check the new variable

Use head() to check that Score_Category has been added. For the rows displayed, check that the labels match the values in Score. Then, using the rules in the code above, identify which categories scores of 50, 51 and 70 would receive. You do not need to find these scores in the displayed rows.

head(lab1_data)
# A tibble: 6 × 5
     ID Group Likeability Score Score_Category
  <dbl> <chr>       <dbl> <dbl> <chr>         
1     1 A               3    55 Medium        
2     2 A               2    72 High          
3     3 A               2    45 Low           
4     4 A               3    50 Low           
5     5 A               2    45 Low           
6     6 A               5    63 Medium

A score of 50 is classified as Low; scores of 51 and 70 are classified as Medium. We will use the original numerical Score variable in our statistical analysis because categorising it discards information.

Describing the data

NoteReminder: populations and samples

A population consists of all the individuals we are interested in studying. A sample is a subset of that population. We usually collect data from a sample because it is impractical to measure everyone in the population.

In the robot study, the 200 participating students form our sample. The population of interest is the wider group of students to whom the researchers want their findings to apply. Descriptive statistics summarise the sample; inferential statistics, such as the t-test, help us assess what the sample results suggest about the population. How widely we can generalise also depends on how the sample was selected.

When we measure a psychological characteristic in a group of people, they will usually obtain different scores. Some students, for example, will obtain higher test scores than others. Together, these observations form a distribution of scores. Descriptive statistics allow us to summarise that distribution rather than having to inspect every score separately.

Centre and spread

You will recall that central tendency describes the centre or typical value of a distribution. The mean is the familiar arithmetic average: add all the scores and divide by the number of scores. The median is the middle value when the scores are arranged in order. Both describe where scores are centred, although the mean is more affected by unusually high or low values.

However, an average does not tell us how similar the individuals are to one another. A class in which everyone scores close to 60 could have the same mean as a class with scores ranging from 40 to 80. This is why we also need measures of variability, or spread: they describe the extent to which scores differ within the distribution.

For our robot study, low variability in test scores would mean that students performed quite similarly; high variability would mean that performance differed more substantially between students. These are individual differences in observed performance.

The summary() function provides a useful first overview:

summary(lab1_data)
       ID            Group            Likeability        Score      
 Min.   :  1.00   Length:200         Min.   :0.000   Min.   :40.00  
 1st Qu.: 50.75   Class :character   1st Qu.:3.000   1st Qu.:53.00  
 Median :100.50   Mode  :character   Median :4.000   Median :59.00  
 Mean   :100.50                      Mean   :3.725   Mean   :59.69  
 3rd Qu.:150.25                      3rd Qu.:5.000   3rd Qu.:66.00  
 Max.   :200.00                      Max.   :7.000   Max.   :88.00  
 Score_Category    
 Length:200        
 Class :character  
 Mode  :character

For a numerical variable, this gives the minimum, first quartile, median, mean, third quartile and maximum. The table below summarises these and some additional measures we will use.

The mean and median are measures of central tendency. The range, interquartile range (IQR), variance and standard deviation (SD) are measures of variability: they describe how spread out the scores are, each in a different way.

Measure What does it tell us? R code for test scores
Mean Sum of the scores divided by the number of scores. mean(lab1_data$Score)
Median Middle score when scores are ordered; for an even number, the mean of the middle two. median(lab1_data$Score)
Minimum and maximum Lowest and highest scores. range(lab1_data$Score)
Range Difference between the highest and lowest scores. diff(range(lab1_data$Score))
Interquartile range (IQR) Difference between the third and first quartiles; describes the spread of the middle 50% of scores. IQR(lab1_data$Score)
Sample variance Measures how widely scores are spread around the mean, using squared deviations. We will work through its calculation below. var(lab1_data$Score)
Sample standard deviation (SD) Measures spread around the mean in the original score units. sd(lab1_data$Score)

The range describes the distance between the two most extreme scores, so it can be strongly affected by a single unusually high or low observation. The IQR focuses on the middle half of the distribution. The first quartile marks roughly the lowest 25% of scores and the third quartile marks roughly the lowest 75%; subtracting them tells us how widely the central 50% are spread. Variance and SD use all the scores and describe their spread around the mean.

The dollar sign selects a variable from a data frame: lab1_data$Score means the Score column in lab1_data. The functions in this table are available in base R or its standard stats package; they do not require dplyr.

WarningWatch out: what range() returns

The statistical range is a single number: maximum minus minimum. R’s range() function returns two numbers, the minimum and maximum. Use diff(range(...)) to calculate their difference.

Let’s calculate several measures of spread for the test scores:

range(lab1_data$Score)
[1] 40 88
diff(range(lab1_data$Score))
[1] 48
IQR(lab1_data$Score)
[1] 13
var(lab1_data$Score)
[1] 106.0259
sd(lab1_data$Score)
[1] 10.29689

Understanding variance and standard deviation

Let’s now examine how variance and SD use the distance of each score from the mean to summarise individual differences.

We first find each score’s deviation from the mean: subtract the mean from the score. A score above the mean has a positive deviation; a score below it has a negative deviation. These deviations sum to zero because the positive and negative values cancel. Squaring the deviations avoids this cancellation and gives greater weight to scores further from the mean.

NoteKey idea: calculating variance and standard deviation

Let’s work through an example using five hypothetical test scores: 46, 48, 50, 52 and 54. Their mean is:

\[ \bar{x} = \frac{46+48+50+52+54}{5} = 50. \]

Subtract the mean from each score, then square each deviation:

Score, \(x_i\) Deviation, \(x_i-\bar{x}\) Squared deviation, \((x_i-\bar{x})^2\)
46 −4 16
48 −2 4
50 0 0
52 2 4
54 4 16
Total 0 40

Adding the squared deviations gives the sum of squares (SS):

\[ SS = \sum_{i=1}^{n}(x_i-\bar{x})^2 = 16+4+0+4+16 = 40. \]

Here, \(n\) is the number of scores, \(\bar{x}\) is their mean, and \(\sum\) tells us to add the squared deviations.

The sum of squares describes the total squared deviation of the scores from their mean. Variance converts this total into an average squared deviation. When calculating sample variance to estimate population variability, we use an adjusted average, dividing by \(n-1\):

\[ s^2 = \frac{SS}{n-1} = \frac{40}{5-1} = 10. \]

The sample variance is 10 points squared.

Why divide by \(n-1\)? When we calculate variance from a sample using \(n\) as the divisor, it tends to underestimate the population variance. Dividing by \(n-1\) instead corrects for this tendency. This is the sample-variance calculation used by R’s var() function.

Finally, we take the square root of the variance to obtain the standard deviation, as R’s sd() function does:

\[ s = \sqrt{10} \approx 3.16\text{ points}. \]

The calculation illustrates why we typically report the standard deviation alongside the mean. The variance is 10 points squared, whereas the SD is approximately 3.16 points. Because SD uses the original test-score units, it is easier to interpret as a measure of how widely students’ scores are spread around the mean.

When comparing variability across groups, we should compare scores measured on the same scale—for example, Alpha’s and Beta’s statistics test scores. It would not be meaningful to directly compare the SD of test scores with the SD of likeability ratings, because test scores are measured on a 0–100 scale and likeability on a 1–7 scale. A larger SD for test scores could therefore reflect the difference in measurement scales rather than greater individual differences.

We can check the example in R:

# Enter the five scores
example_scores <- c(46, 48, 50, 52, 54)

mean(example_scores)  # Mean: 50 points
[1] 50
var(example_scores)   # Sample variance: 10 points squared
[1] 10
sd(example_scores)    # Sample SD: approximately 3.16 points
[1] 3.162278

If all five students had obtained the same score, every deviation would be zero, giving a variance and SD of zero: there would be no individual differences in their observed scores. As scores become more dispersed, variance and SD increase. Neither measure tells us whether average performance is high or low; that is the role of the mean.

TipTry it yourself: the same mean, different variability

Consider these two sets of scores. These are illustrative examples, separate from the robot data:

  • Set A: 48, 49, 50, 51, 52.
  • Set B: 30, 40, 50, 60, 70.

Before calculating anything, predict whether the means differ and which set has the larger variance. Then enter the following code and calculate the mean, variance and SD for each set.

# c() combines values into a vector
scores_a <- c(48, 49, 50, 51, 52)
scores_b <- c(30, 40, 50, 60, 70)

Finally, check that squaring each SD gives its corresponding variance. In R, ^2 means “squared”; for example, 3^2 gives 9.

What does this example tell us about reporting a mean on its own?

mean(scores_a)
[1] 50
mean(scores_b)
[1] 50
var(scores_a)
[1] 2.5
var(scores_b)
[1] 250
sd(scores_a)
[1] 1.581139
sd(scores_b)
[1] 15.81139
# Squaring the SD returns the variance
sd(scores_a)^2
[1] 2.5
sd(scores_b)^2
[1] 250

Both means are 50. Set A has a variance of 2.50 and an SD of approximately 1.58; Set B has a variance of 250.00 and an SD of approximately 15.81. The means alone conceal a substantial difference in variability. Set B’s variance is 100 times larger, while its SD is 10 times larger.

Next week, we will use the idea of variability to understand how ANOVA compares differences between group means with variation among observations within groups.

Checking the data

Descriptive statistics also help us identify possible errors. Let’s examine the minimum and maximum for both dependent variables:

# Minimum and maximum test scores
range(lab1_data$Score)
[1] 40 88
# Minimum and maximum likeability ratings
range(lab1_data$Likeability)
[1] 0 7

Each command selects one variable using $ and returns its minimum and maximum.

TipTry it yourself: identify the problem

Compare the minimum and maximum for Likeability with the response scale described earlier. What is the problem? Why would it be inappropriate to replace the unusual value with 1 simply because 1 is the lowest permitted response?

Then replace range() with sd() to obtain the SD of each variable.

The original data contain a likeability rating of 0, although the response scale runs from 1 to 7. This could reflect a data-entry error or a code for a missing response. We cannot tell which from its value alone, and changing it to 1 would assume that the participant selected the lowest rating.

sd(lab1_data$Score)
[1] 10.29689
sd(lab1_data$Likeability)
[1] 1.344774

In a real study, we would check the original records and coding documentation before deciding how to handle these observations, and document any changes.

For this exercise, we will leave the data unchanged. In a real study, we would resolve this issue before proceeding with the analysis.

Comparing descriptive statistics across groups

So far, our descriptive statistics have combined both robot groups. To address the researchers’ hypotheses, we need separate summaries for Alpha and Beta.

In the next activities, we will focus on statistics test scores (Score), comparing the groups using descriptive statistics, a plot, and an independent-samples t-test. You will then apply the same steps to likeability ratings (Likeability) in the independent practice section.

First, use factor() to tell R that Group represents categories. Setting the levels to A then B also determines the order in which the groups are displayed and compared:

lab1_data$Group <- factor(lab1_data$Group, levels = c("A", "B"))

We can use group_by() from dplyr and get_summary_stats() from rstatix to summarise the scores within each group. We will save the results as score_descriptives so that we can refer to them again:

score_descriptives <- lab1_data %>%
  group_by(Group) %>%  # Calculate separate summaries for Alpha and Beta
  get_summary_stats(Score, show = c("mean", "sd", "min", "max", "iqr"))
score_descriptives     # Display the table
# A tibble: 2 × 8
  Group variable     n  mean    sd   min   max   iqr
  <fct> <fct>    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 A     Score      100  56.6  9.05    40    83    14
2 B     Score      100  62.8 10.6     40    88    15

The output includes the number of observations in each group. Compare the means and SDs: which group scored higher on average, and which was more variable?

get_summary_stats() does not include variance among its standard summaries. We can nevertheless add it to our table using the relationship we have just learned: variance is SD squared. This also gives us another example of creating a variable with mutate():

score_descriptives <- score_descriptives %>%
  mutate(variance = sd^2)  # Add variance by squaring each group's reported SD
score_descriptives         # Display the expanded table
# A tibble: 2 × 9
  Group variable     n  mean    sd   min   max   iqr variance
  <fct> <fct>    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>    <dbl>
1 A     Score      100  56.6  9.05    40    83    14     81.8
2 B     Score      100  62.8 10.6     40    88    15    112.

The added variances are approximate because get_summary_stats() rounds its output. For an unrounded sample variance, use var() on the original scores. For our purpose here, the table makes the relationship between SD and variance clear without introducing a different summary function.

Look at the SD and variance for each group. A larger value means that students’ test scores are more spread out within that group: there are greater individual differences in their observed performance. This is a different question from which robot group obtained the higher mean.

TipTry it yourself: interpreting the difference

Read the mean for each group from the descriptive-statistics table. Subtract Alpha’s mean from Beta’s mean, using a calculator or typing the two numbers into the R Console with a minus sign between them. Express the answer in test-score points.

Does a difference between sample means, on its own, establish a difference between population means?

Beta’s mean minus Alpha’s mean is 62.76 − 56.61 = 6.15 points. Beta also has the larger SD (10.59 compared with 9.05), so its students’ scores are more variable despite their higher average performance.

A positive result means that Beta’s sample mean is higher. However, sample means can differ through sampling variation even when the population means are equal. The size of the observed difference needs to be considered alongside variability and sample size; the t-test will help us do this.

Visualising group differences

A plot helps us inspect the distributions as well as their numerical summaries. We will use ggboxplot() to display the test scores by robot group:

set.seed(5141)  # Reproduce the positions of the jittered points
# Create a box plot using our data
ggboxplot(
  lab1_data,
  # Put robot group on x and the outcome on y
  x = "Group", y = "Score",
  # Colour Alpha blue and Beta red
  color = "Group", palette = c("#3B6B8A", "#AC4141"),
  # Show individual observations with horizontal jitter
  add = "jitter",
  # Label the axes
  xlab = "Robot group", ylab = "Statistics test score",
  # Add a descriptive title
  title = "Test scores by robot group"
)

Box plots and individual test scores for robot groups Alpha and Beta.

The x and y arguments specify the variables plotted on each axis. color = "Group" assigns a different colour to each group, and add = "jitter" displays individual observations with a small horizontal displacement to reduce overlap.

NoteKey idea: reading a box plot

The line inside the box marks the median. The bottom and top of the box mark the first and third quartiles, so its height represents the IQR. The whiskers extend to the most extreme observations within 1.5 IQRs of the corresponding quartile. Observations beyond these limits are flagged separately. Such observations are not automatically errors and should not be removed solely because of their position on a box plot.

TipTry it yourself: interpret the plot

Describe the difference in the location of the two distributions and how much they overlap. How does the plot relate to the group means and SDs? Remember that the line in each box is a median, rather than a mean.

Beta’s scores generally sit higher on the plot, consistent with its higher mean. However, the two distributions overlap considerably. Overlap shows that group membership does not perfectly predict an individual’s score: a difference in group averages does not imply that every person in one group outperforms every person in the other. The plot alone does not establish statistical significance.

Comparing two groups

The independent-samples t-test

Our descriptive statistics and box plot show how the two groups performed in this sample. The next question is whether there is evidence of a difference between the population means. We can examine this using an independent-samples t-test, which compares the means of two separate groups. It is appropriate to use an independent-samples test here because each student worked with only one robot.

The test starts from the null hypothesis that the population means are equal. It asks whether the difference we observed is large enough, relative to the variability in the data and the sample sizes, to provide evidence against that hypothesis.

Let’s run the test using t.test(). In the formula below, Score is our dependent variable and Group identifies the two groups we want to compare:

# Two-sided independent-samples t-test assuming equal population variances
score_test <- t.test(
  Score ~ Group,
  data = lab1_data,
  var.equal = TRUE
)
score_test

    Two Sample t-test

data:  Score by Group
t = -4.4152, df = 198, p-value = 1.658e-05
alternative hypothesis: true difference in means between group A and group B is not equal to 0
95 percent confidence interval:
 -8.896848 -3.403152
sample estimates:
mean in group A mean in group B 
          56.61           62.76

The argument var.equal = TRUE requests the version of the test that assumes equal population variances. We will return to checking assumptions in later sessions.

The main things to look for in the output are:

  • The group means: which robot group scored higher on average?
  • The p-value: is the difference statistically significant? With a significance level of .05, a p-value below .05 leads us to reject the null hypothesis of equal population means.
  • The 95% confidence interval: what range of values does the analysis give for the difference between the population means? This interval is in test-score points, not units of t.

The output also gives the t statistic and degrees of freedom (df), which we need when reporting the result. Earlier, we calculated Beta minus Alpha, giving 6.15 points. Because we ordered the factor levels A then B, R calculates Alpha minus Beta, giving −6.15 points. A negative t therefore indicates a lower mean for Alpha. Reversing the factor order would reverse the sign of t, but its magnitude and the two-sided p-value would remain unchanged.

The t statistic and standard error

The difference between two sample means will vary from sample to sample. The standard error of the mean difference estimates how much this difference would vary if we repeatedly sampled from the same populations. A smaller standard error indicates a more precise estimate.

SD describes how widely individual scores vary around their group mean; the standard error describes uncertainty in our estimate of the difference between the group means.

The t statistic is calculated by dividing the observed mean difference by its standard error. It tells us how many standard errors the observed difference lies from zero—the difference predicted by the null hypothesis.

The degrees of freedom (df) reflect how much independent information is available to estimate variability. Estimating a mean within each group leaves (n-1) degrees of freedom for estimating its variance. For this test, we combine the two groups’ degrees of freedom: (99 + 99 = 198). This value helps determine the reference t distribution used to calculate the p-value.

The p-value

The p-value tells us how likely a result at least as extreme as ours would be if the population means were equal, assuming the test’s assumptions hold. If p < .05, we reject the null hypothesis of equal population means.

The confidence interval

The 95% confidence interval gives a range of plausible values for the difference between the population means. Here, the comparison is Alpha minus Beta, and the interval runs from −8.90 to −3.40 points. This suggests that Alpha’s population mean is between 3.40 and 8.90 points lower than Beta’s. Because the interval does not include zero, it agrees with our conclusion that the difference is statistically significant.

Assumptions

This version of the test assumes that observations are independent, that scores are approximately normally distributed within each population, and that the two populations have equal variances. We will examine these assumptions and how to assess them in later sessions.

Statistical significance does not tell us how large or practically important a difference is. For that, we also need to consider the size of the difference.

Effect size

Cohen’s d expresses the difference between two means in standard-deviation units. To calculate it, we divide the mean difference by a measure of the variability within the groups.

The pooled SD combines the variability within the two groups into a single measure. We use it to express the mean difference in standard-deviation units.

For example, a mean difference of 5 points divided by a pooled SD of 10 points gives d = 0.50: the means differ by half a standard deviation. Let’s calculate Cohen’s d for our robot groups:

score_effect <- cohens_d(
  data = lab1_data,
  formula = Score ~ Group,
  var.equal = TRUE
)
score_effect
# A tibble: 1 × 7
  .y.   group1 group2 effsize    n1    n2 magnitude
* <chr> <chr>  <chr>    <dbl> <int> <int> <ord>    
1 Score A      B       -0.624   100   100 moderate

Because the comparison is Alpha minus Beta, a negative d indicates a lower mean for Alpha. Cohen’s d is not restricted to values between −1 and 1.

Absolute magnitudes of approximately 0.20, 0.50 and 0.80 are often described as small, medium and large, respectively. These are rough conventions; the practical importance of a difference depends on the outcome and research context.

TipTry it yourself: interpreting the result

Use the test and effect-size output to answer the following:

  1. Which group has the higher mean score?
  2. Is the difference statistically significant at the .05 level?
  3. Does the confidence interval include zero?
  4. What does the sign of Cohen’s d indicate?

Beta has the higher mean (62.76 compared with 56.61). The difference is statistically significant, t(198) = −4.42, p < .001. The 95% confidence interval [−8.90, −3.40] does not include zero. Cohen’s d = −0.62: the negative sign indicates that Alpha’s mean is lower than Beta’s, and the means differ by approximately 0.62 pooled standard deviations.

Reporting the findings

A results paragraph should identify the analysis, describe the direction of the difference, and report the relevant descriptive and inferential statistics. Report means and SDs to two decimal places, and identify what the confidence interval refers to.

Use your output to complete the following paragraph:

An independent-samples t-test assuming equal variances was conducted to compare statistics test scores for students assigned Robot Alpha and Robot Beta. Students assigned Robot Alpha (M = …, SD = …) scored [lower/higher] on average than those assigned Robot Beta (M = …, SD = …). The difference was [statistically significant/not statistically significant], t(…) = …, p …, 95% CI […, …], d = ….

Use an exact p-value to three decimal places unless it is below .001, in which case write p < .001. Do not report p = .000. Use the sign of t and d shown in the output.

An independent-samples t-test assuming equal variances was conducted to compare statistics test scores for students assigned Robot Alpha and Robot Beta. Students assigned Robot Alpha (M = 56.61, SD = 9.05) scored lower on average than those assigned Robot Beta (M = 62.76, SD = 10.59). The difference was statistically significant, t(198) = −4.42, p < .001, 95% CI [−8.90, −3.40], d = −0.62.

Your wording need not match this exactly. Check your paragraph against the output:

  • Are each group’s mean and SD attached to the correct robot?
  • Does the direction of the written comparison agree with those means?
  • Have you reported the degrees of freedom, t, p, confidence interval and d?

The confidence interval is in score points, while d is in standard-deviation units. They describe the same comparison on different scales.

Independent practice

Apply the analysis to likeability

TipTry it yourself: adapt the analysis

Using Likeability as the outcome:

  1. Obtain the mean and SD for each robot group.
  2. Produce a box plot with an appropriate title and axis labels.
  3. Run the same equal-variances independent-samples t-test and calculate Cohen’s d.
  4. Draft a results paragraph using the reporting template above.

Use the worked analysis of test scores as a guide, adapting it to the second dependent variable.

Use the code for Score as your starting point. Change the dependent variable wherever it appears, including in the formula and the plot’s y argument. Update labels as well as variable names.

# Summarise likeability by robot group
lab1_data %>%
  group_by(Group) %>%
  get_summary_stats(Likeability, show = c("mean", "sd"))
# A tibble: 2 × 5
  Group variable        n  mean    sd
  <fct> <fct>       <dbl> <dbl> <dbl>
1 A     Likeability   100  2.94  1.15
2 B     Likeability   100  4.51  1.03
# Plot the observed ratings
set.seed(5141)
# Create a box plot using our data
ggboxplot(
  lab1_data,
  # Put robot group on x and the outcome on y
  x = "Group", y = "Likeability",
  # Colour Alpha blue and Beta red
  color = "Group", palette = c("#3B6B8A", "#AC4141"),
  # Show individual observations with horizontal jitter
  add = "jitter",
  # Label the axes
  xlab = "Robot group", ylab = "Likeability rating",
  # Add a descriptive title
  title = "Likeability by robot group"
)

# Compare the groups and calculate the effect size
t.test(Likeability ~ Group, data = lab1_data, var.equal = TRUE)

    Two Sample t-test

data:  Likeability by Group
t = -10.155, df = 198, p-value < 2.2e-16
alternative hypothesis: true difference in means between group A and group B is not equal to 0
95 percent confidence interval:
 -1.874879 -1.265121
sample estimates:
mean in group A mean in group B 
           2.94            4.51 
cohens_d(data = lab1_data, formula = Likeability ~ Group, var.equal = TRUE)
# A tibble: 1 × 7
  .y.         group1 group2 effsize    n1    n2 magnitude
* <chr>       <chr>  <chr>    <dbl> <int> <int> <ord>    
1 Likeability A      B        -1.44   100   100 large

The means are 2.94 for Alpha (SD = 1.15) and 4.51 for Beta (SD = 1.03). An example report is:

Students assigned Robot Alpha (M = 2.94, SD = 1.15) gave lower likeability ratings than students assigned Robot Beta (M = 4.51, SD = 1.03). An independent-samples t-test assuming equal variances indicated that the difference was statistically significant, t(198) = −10.16, p < .001, 95% CI [−1.87, −1.27], d = −1.44.

These values are based on the supplied data, left unchanged as explained in the data-checking exercise.

Additional practice: create a new score band

Complete this exercise if time permits, or use it for further practice after the session.

TipTry it yourself: practising mutate()

Create a variable named Score_Band using the following fictional practice thresholds, which are not the University’s marking scheme:

Score Band
Below 49 Fail
At least 49 but below 59 Pass
At least 59 but below 69 Merit
69 or above Distinction

Check that the new variable has been created. Pay particular attention to scores at the boundaries: 49, 59 and 69.

lab1_data <- lab1_data %>%
  mutate(Score_Band = case_when(
    Score < 49 ~ "Fail",
    Score >= 49 & Score < 59 ~ "Pass",
    Score >= 59 & Score < 69 ~ "Merit",
    Score >= 69 ~ "Distinction"
  ))
head(lab1_data)
# A tibble: 6 × 6
     ID Group Likeability Score Score_Category Score_Band 
  <dbl> <fct>       <dbl> <dbl> <chr>          <chr>      
1     1 A               3    55 Medium         Pass       
2     2 A               2    72 High           Distinction
3     3 A               2    45 Low            Fail       
4     4 A               3    50 Low            Pass       
5     5 A               2    45 Low            Fail       
6     6 A               5    63 Medium         Merit

The boundaries cover every numerical score without gaps or overlap. A score of 49 receives Pass, 59 receives Merit and 69 receives Distinction. Continue to use the numerical Score variable for the analysis.

Before you finish

Save your R script and check that its comments make the sequence of steps clear. Your script should provide a record that you can reopen and run again, starting with loading the packages and importing the data. End your RStudio server session when you have finished.

You should now be able to explain why we report variability alongside a mean, how variance relates to SD, and how descriptive statistics, a plot and a t-test contribute different information about a group comparison.

Next week, we will return to the robot study with three groups. This will introduce one-factor between-participants ANOVA and the distinction between variation within groups and differences between their means.

Back to top