Developing the linear model

PSYC122: Classes in weeks 16-20

• My name is Dr Rob Davies, I am an expert in communication, individual differences, and methods

Targets for weeks 16-19: Concepts

We are working together to develop concepts:

1. Week 16 — Hypotheses, measurement and associations
2. Week 17 — Predicting people using linear models
3. Week 18 — Everything is some kind of linear model
4. Week 19 — The real challenge in psychological science

Targets for weeks 16-19: Skills

We are working together to develop skills:

1. Week 16 — Visualizing, estimating, and reporting associations
2. Week 17 — Using data to predict people
3. Week 18 — Going deeper on linear models
4. Week 19 — Evaluating evidence across multiple studies

Learning targets for this week

• Skills – We need only a limited change to R code
• Concepts – To specify a model with multiple predictors

How we estimate the association between two variables: One outcome and one predictor

model <- lm(mean.acc ~ SHIPLEY, 
            data = clearly.both.subjects)
summary(model)model <- lm(mean.acc ~ SHIPLEY, 
            data = clearly.both.subjects)
summary(model)model <- lm(mean.acc ~ SHIPLEY, 
            data = clearly.both.subjects)
summary(model)

1. Specify the lm function and the model mean.acc ~ ...
2. Specify what data we use data = clearly.both.subjects
3. Get the results summary(model)

How we estimate the association between multiple variables: One outcome and multiple predictors

model <- lm(mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE, 
            data = clearly.both.subjects)
summary(model)model <- lm(mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE, 
            data = clearly.both.subjects)
summary(model)model <- lm(mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE, 
            data = clearly.both.subjects)
summary(model)

4. Specify the lm function and the model:

• mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE

Extensions to the linear model: Multiple predictors

• We assume that the outcome prediction errors residuals are normally distributed
• We do not assume that the distributions of predictor variables are normal

Revision: We typically assume that the residuals are normally distributed

Multiple candidate predictor variables

Figure 4: Scatterplots showing the potential association between accuracy of comprehension and variation on each of a series of potential predictor variables. Data are from two studies

We do not assume normal predictors

Figure 5: Grid of plots showing the distribution of potential predictor variables

Extensions to the linear model: Multiple predictors

Analyses are done in context so when we conduct analyses we must use contextual information

1. We want to know: What makes it easy or difficult to understand written health information?
2. So our research questions include:

Given theory, a model of comprehension accuracy should include – as predictors:

(1.) experience HLVA, SHIPLEY and (2.) reasoning ability (FACTOR3, reading strategy) (Freed et al., 2017)

The flexibility and power of linear models requires us to be aware of the garden of forking paths

Different researchers can reasonably make different choices

This is why we teach:

• Evidence reviews like meta-analysis \(\rightarrow\) theory- and evidence-based selection of critical variables for analysis
• Open science: we share usable data and analysis code in open repositories \(\rightarrow\) so others can critically evaluate our work

Coding the linear model with multiple predictors

lm(mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE, ...)

• The code represents a linear model with multiple predictors:
• \(y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \dots + \epsilon\)

Thinking about the linear model with multiple predictors

\(y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \dots + \epsilon\)

Outcome \(y\) is calculated as the sum of:

• The intercept \(\beta_0\) plus
• The product of the coefficient of the effect of e.g. AGE \(\beta_1\) multiplied by \(x_1\) a person’s age +
• + any number of other variables +
• The error \(\epsilon\): mismatches between observed and predicted outcomes

Identifying key information in results

1. The summary() of the linear model shows:
2. Estimates of the coefficients of the effects of the predictors we included, with null hypothesis significance tests of those estimates
3. Model fit statistics including R-squared and F-statistic estimates

Plot predictions to interpret effects

Figure 8: A grid of plots showing model predictions, for outcome accuracy, given variation in (a.) age, (b.) vocabulary, (c.) health literacy, and (d) reading strategy

Critically evaluating the results of analyses involving linear models

There are three levels of uncertainty when we look at sample data (McElreath, 2020) – uncertainty over:

1. The nature of the expected change in outcome
2. The ways that expected changes might vary between individual participants or between groups of participants
3. The random ways that specific responses can be produced

Critically evaluating the results of analyses involving linear models

• These uncertainties require us to carefully qualify the conclusions we draw from data analyses
• This does not mean we should avoid causal language when we think that psychological processes cause the behaviours we examine (Grosz et al., 2020)
• But it does mean we can be careful to identify the limits in the evidence we analyse

Revision: As we move into thinking about the data analysis, we need to identify our assumptions

1. validity: that differences in knowledge or ability cause differences in test scores
2. measurement: that this is equally true across the different kinds of people we tested
3. generalizability: that the sample of people we recruited resembles the population

How do you do this work?

1. validity

• We want to work with valid measures but validity requires explaining (Borsboom et al., 2004):

1. Does the thing exist in the world?
2. Is variation in that thing be reflected in variation in our measurement?

• What you can do: literature review \(\rightarrow\) to identify your reasoning in answer to these questions

How do you do this work?

2. measurement
3. generalizability

• It is most helpful to assume from the start that effects estimates will vary (a. Gelman, 2015; Vasishth & Gelman, 2021)
• So then we ask ourselves: will this test work in the same way in different groups?
• And we ask: how will these effects estimates vary across different groups

Why we need replication studies

Figure 9: Scatterplot showing the potential association between accuracy of comprehension and vocabulary scores: Data from eight studies. Effects will vary between different samples so: expect the variation (a. Gelman, 2015; Vasishth & Gelman, 2021) >>> important to evaluating claims in the literature, and to evaluation of your own results

Why we need to consider the generalizability of sample data

Figure 10: Grid of plots showing the distribution of potential predictor variables

Convenience samples are common in Psychology

• We test who we can – convenience sampling – and who we can test has an impact on the quality of evidence (Bornstein et al., 2013)

Everything is some kind of linear model

• Most introductory statistics classes teach each statistical test as if they are independent

The t-test as linear model

\(y_i = \beta_0 + \beta_1X\)

• If you have two groups, with a variable X coding for group membership
• Then the mean outcome for one group \(= \beta_0\)
• The estimate of the slope \(\beta_1\) tells about the average difference between groups
• And we can code the model like this: lm(y ~ group)

Extensions to the linear model

\(outcome ~ predictors + error\)

• outcome can generalize to analyse data that are not metric, do not come from normal distributions
• predictors can be curvilinear, categorical, involve interactions
• error can be independent; can be non-independent

Extensions to the linear model – binary or dichotomous outcomes

1. Binary outcomes are very common in Psychology: yes or no; correct or incorrect; left or right visual field etc.
2. The change in coding is e.g. glm(ratings ~ predictors, family = "binomial")

Extensions to the linear model – non-independence of observations

1. Much – maybe most – psychological data are collected in ways that guarantee the non-independence of observations

• We test children in classes, patients in clinics, individuals in regions
• We test participants in multiple trials in an experiment, recording responses to multiple stimuli

2. These data should be analysed using linear mixed-effects models (Meteyard & Davies, 2020) which we study at MSc

General advice

An old saying goes:

All models are wrong but some are useful

(attributed to George Box).

Summary

• Linear models are a very general, flexible, and powerful analysis method
• We can use assuming that prediction outcomes (residuals) are normally distributed
• With potentially multiple predictor variables

Summary

• When we plan an analysis we should try to use contextual information – theory and measurement understanding – to specify our model
• When we critically evaluate our or others’ findings, we should consider validity, measurement, and generalizability

Summary

• When we report an analysis, we should report:

1. Explain what I did, specifying the method (linear model), the outcome variable (accuracy) and the predictor variables (health literacy, reading strategy, reading skill and vocabulary)
2. Report the model fit statistics overall (\(F, R^2\))
3. Report the significant effects (\(\beta, t, p\)) and describe the nature of the effects (does the outcome increase or decrease?)