Developing the linear model

PSYC411: Classes weeks 6-10

• My name is Dr Rob Davies, I am an expert in communication, individual differences, and methods

Week 10: Developing the linear model

Figure 1: Scatterplot showing the potential association between accuracy of comprehension and vocabulary scores: Data from eight studies

Analyze + visualize + present

Develop the linear model: our aims

• We will learn how to:

1. Extend our capacity to code models so that we can incorporate multiple predictors
2. Develop the thought processes required to make decisions about what predictors to include
3. Develop the skills required to critically evaluate results

• Especially considering potential variation across samples

Develop the linear model: our aims

• We will revise how to:

5. Identify and interpret model statistics
6. Critically evaluate the results
7. Communicate the results

• We will learn how to: explore extensions of the linear model

We close the loop: Our context, the health comprehension project

We close the loop: Health comprehension project, questions and analyses

1. We want to know: What makes it easy or difficult to understand written health information?
2. So our research questions are:

• What person attributes predict success in understanding?
• Can people accurately evaluate whether they correctly understand written health information?

Extensions to the linear model: Multiple predictors

• We need only a limited change to R code
• To specify a model with multiple predictors

How we estimate the association between two variables: One outcome and one predictor

model <- lm(mean.acc ~ SHIPLEY, 
            data = all.studies.subjects)
summary(model)

1. Specify the lm function and the model mean.acc ~ ...
2. Specify what data we use data = all.studies.subjects
3. Get the results summary(model)

How we estimate the association between multiple variables: One outcome and multiple predictors

model <- lm(mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE, 
            data = all.studies.subjects)
summary(model)

4. Specify the lm function and the model:

• mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE

The sentence structure of model code in R

Take a good look:

lm(mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE, ...)

You will see this sentence structure in coding for many different analysis types

• method(outcome ~ predictors)
• predictors could be SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE ...

Extensions to the linear model: Multiple predictors

• We assume that the outcome prediction errors residuals are normally distributed
• We do not assume that the distributions of predictor variables are normal

Revision: We typically assume that the residuals are normally distributed

• Some outcome prediction errors – residuals – are positive
• Some residuals are negative
• The average of the residuals will be zero overall

Multiple candidate predictor variables

Figure 5: Scatterplot showing the potential association between accuracy of comprehension and variation on each of a series of potential predictor variables. Data from 8 studies

We do not assume normal predictors

Figure 6: Grid of plots showing the distribution of potential predictor variables. Data from 8 studies

Extensions to the linear model: Multiple predictors

Analyses are done in context so when we conduct analyses we must use contextual information

Closing the loop: The health comprehension project questions

We must use contextual information: theory of comprehension

Given theory, model of comprehension accuracy should include measures of

(1.) experience (HLVA, SHIPLEY) and (2.) reasoning ability (reading strategy)

The flexibility and power of linear models requires us to be aware of the garden of forking paths

Different researchers can reasonably make different choices

This is why we care about open science

• Theory- and evidence-based selection of critical variables for analysis \(\rightarrow\) literature review
• Share usable data and analysis code in open repositories \(\rightarrow\) research report exercise, PSYC403 data archiving

Let’s take a break

Coding, thinking about, and reporting linear models with multiple predictors

lm(mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE, ...)

Coding the linear model with multiple predictors

lm(mean.acc ~ SHIPLEY + HLVA + FACTOR3 + AGE + NATIVE.LANGUAGE, ...)

• The code represents a linear model with multiple predictors:
• \(y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \dots + \epsilon\)

Thinking about the linear model with multiple predictors

\(y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \dots + \epsilon\)

Outcome \(y\) is calculated as the sum of:

• The intercept \(\beta_0\) plus
• The product of the coefficient of the effect of e.g. AGE \(\beta_1\) multiplied by \(x_1\) a person’s age +
• + any number of other variables +
• The error \(\epsilon\): mismatches between observed and predicted outcomes

Identifying key information in results

1. The summary() of the linear model shows:
2. Estimates of the coefficients of the effects of the predictors we included, with null hypothesis significance tests of those estimates
3. Model fit statistics including R-squared and F-statistic estimates

Plot predictions to interpret effects

Figure 10: A grid of plots showing model predictions, for outcome accuracy, given variation in (a.) age, (b.) vocabulary, (c.) health literacy, (d) reading strategy and (e.) native language. Data from eight studies

Let’s take a break

Critically evaluating the results of analyses involving linear models

There are three levels of uncertainty when we look at sample data (McElreath, 2020) – uncertainty over:

1. The nature of the expected change in outcome
2. The ways that expected changes might vary between individual participants or between groups of participants
3. The random ways that specific responses can be produced

Critically evaluating the results of analyses involving linear models

• These uncertainties require us to carefully qualify the conclusions we draw from data analyses
• This does not mean we should avoid causal language when we think that psychological processes cause the behaviours we examine (Grosz et al., 2020)
• But it does mean we can be careful to identify the limits in the evidence we analyse

Revision: As we move into thinking about the data analysis, we need to identify our assumptions

1. validity: that differences in knowledge or ability cause differences in test scores
2. measurement: that this is equally true across the different kinds of people we tested
3. generalizability: that the sample of people we recruited resembles the population

How do you do this work?

1. validity

• We want to work with valid measures but validity requires explaining (Borsboom et al., 2004):

1. Does the thing exist in the world?
2. Is variation in that thing be reflected in variation in our measurement?

• What you can do: literature review \(\rightarrow\) to identify your reasoning in answer to these questions

How do you do this work?

2. measurement
3. generalizability

• It is most helpful to assume from the start that effects estimates will vary (Gelman, 2015; Vasishth & Gelman, 2021)
• So then we ask ourselves: will this test work in the same way in different groups?
• And we ask: how will these effects estimates vary across different groups

Why we need replication studies

Figure 11: Scatterplot showing the potential association between accuracy of comprehension and vocabulary scores: Data from eight studies. Effects will vary between different samples so: expect the variation (Gelman, 2015; Vasishth & Gelman, 2021) >>> important to evaluating claims in the literature, and to evaluation of your own results

Why we need replication studies

Figure 12: Effects will vary between samples so expect the variation (Gelman, 2015; Vasishth & Gelman, 2021) >>> ask what variation may result from systematic differences between groups

Why we need to consider the generalizability of sample data

Figure 13: Grid of plots showing the distribution of potential predictor variables

Convenience samples are common in Psychology

• We test who we can – convenience sampling – and who we can test has an impact on the quality of evidence (Bornstein et al., 2013)
• If age, ethnicity or gender are not balanced \(\rightarrow\) does this matter to your research question?
• If samples are limited in size \(\rightarrow\) how does that affect our uncertainty over effects estimates?

Let’s take a break

The linear model is very flexible, powerful and general

• Most introductory statistics classes teach each statistical test as if they are independent

The t-test as linear model

\(y_i = \beta_0 + \beta_1X\)

• If you have two groups, with a variable X coding for group membership
• Then the mean outcome for one group \(= \beta_0\)
• The estimate of the slope \(\beta_1\) tells about the average difference between groups
• And we can code the model like this: lm(y ~ group)

Extensions to the linear model

\(outcome ~ predictors + error\)

• outcome can generalize to analyse data that are not metric, do not come from normal distributions
• predictors can be curvilinear, categorical, involve interactions
• error can be independent; can be non-independent

Look ahead: extensions to the linear model

• What if the outcome measurement data cannot be understood to be metric or to come from a normal probability distribution?

Extensions to the linear model – binary or dichotomous outcomes

1. Binary outcomes are very common in Psychology: yes or no; correct or incorrect; left or right visual field etc.
2. The change in coding is e.g. glm(ratings ~ predictors, family = "binomial")

Extensions to the linear model – ordinal outcomes

1. Likert scale or ratings data are best analysed using ordinal models (Liddell & Kruschke, 2018)
2. The change in coding (see Christensen, 2022) is e.g. clm(ratings ~ predictors)

Extensions to the linear model – non-independence of observations

1. Much – maybe most – psychological data are collected in ways that guarantee the non-independence of observations

• We test children in classes, patients in clinics, individuals in regions
• We test participants in multiple trials in an experiment, recording responses to multiple stimuli

2. These data should be analysed using linear mixed-effects models (Meteyard & Davies, 2020)

General advice

An old saying goes:

All models are wrong but some are useful

(attributed to George Box).

Summary

• Linear models are a very general, flexible, and powerful analysis method
• We can use assuming that prediction outcomes (residuals) are normally distributed
• With potentially multiple predictor variables

Summary

• Closing the loop: when we plan an analysis we should try to use contextual information – theory and measurement understanding – to specify our model
• Closing the loop: when we critically evaluate our or others’ findings, we should consider validity, measurement, and generalizability

Summary

• When we report an analysis, we should report:

1. Explain what I did, specifying the method (linear model), the outcome variable (accuracy) and the predictor variables (health literacy, reading strategy, reading skill and vocabulary)
2. Report the model fit statistics overall (\(F, R^2\))
3. Report the significant effects (\(\beta, t, p\)) and describe the nature of the effects (does the outcome increase or decrease?)

End of week 10