Research report – why?

PSYC411: Classes weeks 6-10

• My name is Dr Rob Davies, I am an expert in communication, individual differences, and methods

PSYC411: Classes weeks 6-10

We are working together to develop skills in a series of classes

1. Week 6 — Research report
2. Week 7 — Associations, hypotheses
3. Week 8 — Linear models: predictions
4. Week 9 — Visualizing data
5. Week 10 — Linear models: development

This week: we focus on the research report, especially why

• Why: what is the motivation for the assignment?
• Explain this, to help to make sense of what we are doing, and what you can learn

How to work with the materials

• Watch these lectures for an outline
• Read the notes to get a detailed explanation
• Follow references to sources to build expertise

What to do and how to do it: see notes, and discuss in class

Why this? Key ideas

1. The difference between science “…being done, science in the making, and science already done, a finished product …” [Bourdieu (2004); p.2]
2. Published reported analyses are not necessary or sufficient to the data or the question

Wider context: Credibility revolution

Science (including psychological science) has undergone a rolling series of crises:

• Replicability or replication crisis (Pashler & Wagenmakers, 2012)
• Statistical crisis (Gelman & Loken, 2014)
• Generalizability crisis (Yarkoni, 2022)

Wider context: Credibility revolution

Nosek et al. (2022): replication outcomes for three systematic replication studies

The triggers for crisis

• Failures to replicate influential claims (Nosek et al., 2022)
• Questionable research practices (John et al., 2012)
• Questionable measurement practices (Flake & Fried, 2020)
• Limited samples (Button et al., 2013; Henrich et al., 2010; Wild et al., 2022)

The credibility revolution: responses

• Pre-registration (Nosek et al., 2018, 2019) and registered reports (Nosek & Lakens, 2014)
• Replication studies (e.g. Aarts et al., 2015)
• Identification of open science principles (Munafò et al., 2017)

What is the motivation?

Opportunity to do original research work

Let’s take a break

• End of part 1

Multiverses: a fresh perspective

• We introduce the idea that your analysis work will flow through the stages of a pipeline
• From getting the data |> to presenting your findings
• Because, next, we examine how pipelines can multiply

Data analysis: we get from raw data to presenting results in stages

The data analysis pipeline (workflow)

• Get some data
• Process or tidy the data
• Explore, visualize, and analyze the data
• Present or report your findings

The garden of forking paths

The garden of forking paths: pathways multiply at each choice

• It is possible to have multiple potential different paths from the data to the results
• The results may vary, depending on the path we take

Credibility and multiplicity

• Certainty over data processing or the uncertainty over analysis methods revealed in multiverse analyses (next)
• Plus limitations of data and code sharing, and incompleteness of reports
• Drives problems replicating or reproducing claims

Multiple pathways and secret lives

Carp (2012): Proportions of studies that reported using each of 21 procedures for data collection and analysis.

Data processing is always necessary

• When you collect or access data for a research study
• The complete raw dataset you receive is almost never the complete dataset you analyze or whose analysis you report

Data multiverse analyses

Data multiverse: lessons

• We approach our study with the same research question, and prediction
• We begin with the exact same data
• We could construct different datasets depending on equally reasonable choices
• As a result, we may see different results for the analyses of the different datasets

Analysis multiverse

Different researchers may adopt different approaches (Silberzahn & Uhlmann, 2015)

Silberzahn and Uhlmann (2015): Twenty-nine research teams reached a wide variety of conclusions using different methods on the same data set to answer the same question (about football players’ skin colour and red cards).

Analysis multiverse

In multiverse analyses, we typically see variation in:

• how psychological constructs are operationalized;
• how data are processed or datasets constructed;
• what statistical techniques are used;
• how those techniques are used;
• \(\rightarrow\) with associated variation in results

Conclusions

If we see one analysis reported in a paper that does not mean only one analysis can reasonably be applied

Let’s take a break

• End of Part 2

Kinds of reproducibility

(Gilmore et al., 2017; following Goodman et al., 2016) present three kinds of reproducibility:

• Methods reproducibility
• Results reproducibility
• Inferential reproducibility

We are mostly focused — in thinking about the research report — on methods and inferential reproducibility

Methods reproducibility

• Other researchers should be able to get the same results if they use the analysis methods with the same data

Hardwicke et al. (2021): Frequency of reproducibility outcomes by value type.

Inferential reproducibility

If researchers repeat a study (results reproducibility) or re-analyze original data (methods reproducibility) then they should come to similar conclusions as original authors

But …

• reproducibility attempt could reveal problems, uncertainty over choices
• different researchers could apply different prior expectations over the probability of possible effects

The match between open science ideas and practices

The lessons learned from crises mean we now hope to see researchers:

• Share data and code;
• Publish research reports in ways that enable others to check or query analyses.
• What do we see, when we look at current practices?

Data and code sharing

• Analyses by Kidwell et al. (2016) and analyses reviewed by Nosek et al. (2022): study data increasingly available

Nosek et al. (2022): Yearly counts of users, sharing of files (research data, materials, code), and registration of studies on OSF and AsPredicted.

Data and code sharing

It is great that data are shared but analyses show they are not always readily usable (Towse et al., 2021) but should be

• completeness: are all the data and the data descriptors supporting a study’s findings publicly available?
• reusability: how readily can the data be accessed and understood by others?

Enabling others to check or query analyses

Overall, the good news is

• for many published reports, if data are shared and if the shared data are accessible and reusable then
• most of the time, the researchers can reproduce the results presented by the original study authors (Hardwicke et al., n.d.; Hardwicke et al., 2018; Laurinavichyute et al., 2022; Minocher et al., n.d.; Obels et al., 2020; but see Crüwell et al., n.d.)

Yet studies of reproducibility identify challenges

Data challenges

1. Data Availability Statements or open science badges indicate data are shared but data may not be directly accessible
2. The data are shared and accessible but there may be missing or incorrect information about the data
3. Original study authors may share raw and processed data or just processed or just raw data
4. There may be mismatches between the variables referred to in the report and the variables named in the data file

Analysis challenges

1. The description of the analysis procedure may be incomplete or ambiguous
2. It may be difficult to identify the key analysis
3. It is easier to reproduce results if both data and code are shared but code is not always shared
4. Sometimes, analysis code is shared but it is difficult to use
5. Sometimes, there are errors in the analysis

Summary: why this?

1. We are in the middle of a credibility revolution
2. Focusing on data analysis, it is useful to think about the whole data pipeline
3. At every stage of the data pipeline, there are choices: forking paths
4. We can share data and enable others to check, query analysis choices

Opportunities

• The conventions of normal scientific practice can work so that if we encounter an anomaly we blame ourselves (Kuhn, 1970)
• Sometimes, if you think you have found an error or a problem in a published analysis or a shared dataset, remember: you may be right (Herndon et al., 2014)

What we learn: critical reading skills

• Learn to critically read research reports and data documentation
• Develop proficiency in analysis workflow
• Develop critical skills in analysis

What we learn: good practice sense

• The credibility revolution requires us to think about and to teach good open science practices that safeguard the value of evidence in psychology

What we learn: practical sense

• People make mistakes
• Different choices are often reasonable
• We always need to check the evidence

What we learn: do better

• Share data and code
• Publish research reports in ways that enable others to check or query analyses