Our approach: Concepts, skills, levels

• Each week, we focus on building concepts and skills
• In conceptual work, we aim for deep and broad understanding of what you do and why
• In practical work, we introduce, consolidate, or extend within a single problem set to grow your independence

The new idea: data analysis in context

• Traditionally, psychologists have to teach a different procedure each week: a-test-a-week
• limited discussion of theory or measurement,
• focus on rules about doing and reporting null hypothesis significance tests
• but:

1. This risks student (researcher) focus on doing the significance test (when, what, how)
2. While the real challenges are located in figuring out what we want to find out, measure, and explain

Targets for weeks 16-19: Concepts

We are working together to develop concepts:

1. Week 16 — Hypotheses, measurement and associations
2. Week 17 — Predicting people using linear models
3. Week 18 — Everything is some kind of linear model
4. Week 19 — The real challenge in psychological science

Targets for weeks 16-19: Concepts

Targets for weeks 16-19: Skills

We are working together to develop skills:

1. Week 16 — Visualizing, estimating, and reporting associations
2. Week 17 — Using data to predict people
3. Week 18 — Going deeper on linear models
4. Week 19 — Evaluating evidence across multiple studies

Targets for weeks 16-19: Skills

The benefits: critical thinking and you

• To get a B+ or an A you will need to show critical reflection
• Our work here will build your ability to do this

Statistical rituals largely eliminate critical thinking

• Traditionally, students learn statistical tests, and learn to identify if a test statistic is significant or not
• If we do not also talk about what is actually observed, and whether or how it is or is not compatible with theory-based predictions then we do ritual not science (Gigerenzer, 2004)
• This is a problem: the focus on significance allows us to build or accommodate vague theories that can never be wrong

Open, reproducible, methods are not enough

• Now: we need to think causally about predictions and measurement

1. We need better theory so we can build clear testable predictions from explicit assumptions
2. We need better measurement because if we cannot reliably measure something then it is hard to build a theory about it

We need to think about the derivation chain

Here’s a toolkit for thinking productively about your hypotheses

The derivation chain (Meehl, 1990; Scheel et al., 2021)

1. Develop your theory: the concepts, and the assumptions about causality
2. Specify how psychological concepts will be measured
3. Identify auxiliary assumptions about how we get from theoretical concepts to observable data
4. Identify theoretical predictions
5. Link theoretical predictions to specific statistical tests that may support or contradict them

Valid measures

• We often teach and learn about different kinds of validity but the key idea is simple (Borsboom et al., 2004):

  a test is valid for measuring an attribute if and only if (a) the attribute exists and (b) variations in the attribute causally produce variations in the outcomes of the measurement procedure

• We want to work with valid measures but validity requires explaining: (Q.1) Does the thing exist in the world? (Q.2) Is variation in that thing be reflected in variation in our measurement?

Summary: our critical thinking checklist

• What is our (causal) theory?
• What measures are we using, why?
• What is our specific prediction, why?
• Does the prediction relate to sign and to magnitude?
• What analysis can test this prediction, why?
• How will our results affect our beliefs, why?

Let’s take a break

Learning targets for this week:

• Concepts: begin learning to think critically
• Skills: identify how we build hypotheses

Case study: the health comprehension project

Why this? We don’t really know what makes it easy or difficult to understand advice about health

Health comprehension project: impacts

• We are working to improve health communication
• With partners at Vienna Business University, Kantar Public, and the London School of Economics
• Our results could change: business and health communication; understanding reading development

Health comprehension project: questions and analyses

• Our research questions are:

• These kinds of research questions can be answered using methods like correlation, linear models

Health comprehension project – relevance: methods you will use in your professional work

• We collect data using online Qualtrics questionnaire surveys
• We test people on a range of dimensions using standardized ability and our own knowledge tests
• Many of you will go on to work with online surveys, and with data from standardized ability measures

You can get involved

Health comprehension project: why it is a case study

Cognitive process theory of comprehension success

• When skilled adult readers read and try to understand written text (Kintsch, 1994)
• They must recognize and access the meanings of words
• Then use knowledge and reasoning to build an interpretation of what is in the text
• Based on connecting the information in the text with what they already know

Individual differences theory of comprehension success

• Successfully understanding text depends on (1.) language experience and (2.) reasoning ability (Freed et al., 2017)

Where the data come from: our measures

• We measure reading comprehension: asking people to read text and then answer multiple choice questions
• We measure background knowledge: vocabulary knowledge (Shipley); health literacy (HLVA)
• We ask people to rate their own understanding of each text

Example critical evaluation questions

• Are multiple choice questions good ways to probe understanding? – What alternatives are there?
• Are tests like the Shipley good measures of language knowledge? – What do we miss?
• Can a person accurately evaluate their own understanding? – Can we rely on subjective judgments?

Relevance to you

• Here, we are looking ahead to the critical thinking you will need to show in your second and third year essays

Let’s take a break

Learning targets for this week:

• Concepts – associations: correlations, estimates and hypothesis tests
• Skills – visualizing variation and covariation
• Skills – writing the code
• Skills – estimating correlations
• Skills – interpreting and reporting correlations

Talking about the relationships between variables

• Psychologists and people who work in related fields often want to know about associations
• Is variation in observed values on one dimension (e.g., comprehension) related to variation in another dimension (e.g., vocabulary)?
• Do values on both dimensions vary together?

The language in this area can vary: we will be consistent but you need to be aware of the different terms

• Outcome \(=\) response \(=\) criterion \(=\) dependent variable
• Predictor \(=\) covariate \(=\) independent variable \(=\) factor
• Linear model \(=\) regression analysis \(=\) regression model \(=\) multiple regression

Let’s look at the data we will use

• The person in row 1 has ETHNICITY White, is AGE 34 years, scored 33 on Shipley vocabulary, scored 7 on HLVA health literacy
• and, on average, self-rated their understanding of health information as 7.96 (so 8/9, mean.self)
• while scoring 0.49 accuracy in tests of understanding (49% mean.acc)

# A tibble: 4 × 6
  mean.acc mean.self  HLVA SHIPLEY   AGE ETHNICITY
     <dbl>     <dbl> <dbl>   <dbl> <dbl> <fct>    
1     0.49      7.96     7      33    34 White    
2     0.85      7.28     7      33    25 White    
3     0.82      7.36     8      40    43 White    
4     0.94      7.88    11      33    46 White    

Destination correlation: where the correlation number comes from

Covariance

\[COV_{xy} = \frac{\sum(x - \bar{x})(y - \bar{y})}{n -1}\]

• If we want to estimate the correlation between two sets of numbers: \(x\) and \(y\)
• We want to know if variation in \(x\) (given by \(x - \bar{x}\))
• Varies together with variation in \(y\) (given by \(y - \bar{y}\))

Destination correlation: where the correlation number comes from

Covariance divided by standard deviations

\[r = \frac{COV_{xy}}{s_xs_y}\]

• Because the two sets of numbers can be on different scales: e.g., SHIPLEY out of 40; mean.acc (proportion, out of 1)
• And because covariance values depend on the scales
• To make correlations easier to compare, we remove scaling by dividing by the standard deviations of the variables

Let’s think about an example

• Measurement: Someone with higher scores on tested accuracy of understanding will also present higher scores on their ratings of their own understanding
• Statistical prediction: We predict that mean.acc and mean.self scores will be associated
• Test: If the prediction is correct, mean.acc and mean.self scores will be correlated

Distributions: Let’s see what this means – how do scores vary?

Histograms showing the distribution of mean accuracy and mean self-rated accuracy scores in the ‘clearly.one.subjects’ dataset: means calculated for each participant over all their responses

A histogram is a useful way to show the distribution of values

When we talk about variance we are talking about how values vary in relation to the mean for the sample

We are talking about how values vary in relation to the mean for the sample

The basic question when we examine covariance: do values vary together?

• If the person at row 1 has a mean.accuracy score of .49, lower than the average
• And the person at row 4 has a mean.accuracy score of .94, higher than the average
• What will their mean.self scores be: will they be higher or lower than the average mean.self score?

We can use scatterplots to examine associations

Scatterplots showing whether values on mean accuracy (mean.acc) vary together with values on mean self-rated accuracy (mean.self) for the participants in this sample

Let’s take a break

The R code for a correlation test, bit by bit

cor.test(clearly.one.subjects$mean.acc, 
         clearly.one.subjects$mean.self,
         method = "pearson")

Reporting a correlation

Mean accuracy and mean self-rated accuracy were significantly correlated (\(r (167) = .49, p < .001\)). Higher mean accuracy scores are associated with higher mean self-rated accuracy scores.

What will different kinds of correlations look like?

We can simulate data to demonstrate: (left) the correlation is positive, \(r = .5\); (right) the correlation is negative, \(r = -.5\)

Scatterplots showing how simulated data values on mean accuracy and mean self-rated accuracy could vary together given positive or negative correlations

We can also imagine – again with simulated data – what correlations of increasing size might look like

Scatterplots showing how simulated data values on mean accuracy and mean self-rated accuracy could vary together given positive correlations of increasing size

Summary

• We are often interested in whether or how variation in the values of two variables are associated
• We can visualize the distribution of values in any one variable using histograms
• We visualize the association of values in two variables using scatterplots
• We conduct correlation tests to examine the sign (positive or negative) and the strength of the association
• But we always need to think about our research questions, about where our data come from and about whether our measures are any good

End of lecture