Being a scientist
doing a science

We are learning how to think critically for ourselves

Active learning works best even if it sometimes feel challenging

• Active learning is what you do when you ask or answer questions
• We will prompt you sometimes with Activate

What we are going to do

We are going to engage with big ideas and difficult questions

1. Being a psychologist: How should we do science, how do we do science?
2. How we try not to fool ourselves: The logic of the scientific method, and how we should use it
3. Why this is harder in Psychology: People are semi-random, words are slippery, we can’t look inside everyone’s heads
4. Science as a thing that is done by people: History, community, inequity, and diversity

Big ideas and difficult questions

• What is science?
• How do we do it?

Science versus pseudoscience

We are surrounded by pseduoscientific claims and it’s hard to know what to believe. What is the difference between

1. Scientific claims like: phonics are a good way to learn to read
2. Pseudoscience claims like: vaccination causes Autism

Am I a crank or what?

• The problem is that pseudoscience can be presented to look like science and cranks can talk like scientists
• How do we tell the difference?
• Bright & Heesen (2023): the difference does not lie in whether we are right or wrong, doing good science or bad
• The difference lies in communism (Merton, 1942)

Communism, universalism, disinteredness, organized skepticism

Merton (1942) sees the ethos of modern science as including:

• Communism: make your work freely available to others, and open to evaluation
• Universalism: evaluate ideas independent of who their sources are
• Disinterestedness: evaluate ideas based on evidence or reason, not preferences or interests
• Organised skepticism: suspension of judgment, openness to being wrong

Doing science versus bullshitting

The difference between science and pseudoscience is like the difference between liars and bullshitters (Hicks et al., 2024):

• Liars give you a statement they know to be false, with the intention to deceive you
• Bullshitters do not care about the truth of their statements, whether or not they intend to deceive you about what they think

Careful observations organized to enable reasoning

If we do care about the truth, how do we get to it?

• We have, at least since the Greek philosopher Aristotle, thought that we needed careful observations organized so that we can think effectively reason about them

What will our reasoning involve?

Most of the time, the forms of reasoning we use go in one of two directions

1. Induction \(\rightarrow\) From observations to fundamental and general insights, regularities or principles
2. Deduction \(\rightarrow\) To observations or tests, from fundamental and general insights, regularities or principles

Being capable of being wrong

Maybe the core idea many psychological scientists assume about our work as scientists is that we must be capable of being wrong: roughly, this involves

1. Assuming a theory about the world or about people
2. Specifying a prediction that the theory implies, and testing it

The history of science

• In thinking about science, it is useful to keep in mind the difference between (1.) what we think people should do (Merton, 1942; Popper, 2005) and (2.) what we actually do
• This leads us to think about history or the sociology of science in ways that are enlightening, especially when we want to engage with messy realities

The normal business of science: solving puzzles

• Most of the time, scientists, working as members of a community or a tradition, work within a shared understanding of:

  • what problems are worth investigating
  • what solutions are possible
  • what methods can be used

• In the context of this paradigm (Kuhn, 1970), the scientist can spend their time figuring out solutions to the uncertainties that remain: solving puzzles

Anomalies and revolutionary change

But over time scientists may come to see a (Kuhn, 1970):

… persistent failure of the puzzles of normal science to come out as they should

A time of crisis ensues, leading to rejection of one paradigm, and acceptance of another

Big ideas and difficult questions

The hardest science: Why learning about people is hard

You will learn about revolutionary new ways of doing psychological science

• We will talk about

  • open science practices
  • reproducibility
  • requirements for building credible knowledge

• But why this revolution became necessary is worth thinking about

The hardest science: Why learning about people is hard

Srivastava (2009) argues:

• Hard problems are embedded in complex systems

• Hard problems vary by local conditions

• Hard problems are difficult to quantify

  • This is why Psychology is so challenging and so exciting

People vary, diverse in who we are, what we think and how we behave

Figure 1: The people in any group will vary in many ways: just look around you

We can never test all the people on all the things

Figure 2: Normally, when we do psychological studies, we ask a sample of people to respond to a sample of stimuli

People vary in how they respond

Figure 3: Different people will often respond in different ways: from strongly negative (dark blue) through non-responsive (near-white) to strongly positive (dark red)

We will work with data but numbers impose limits

We often work with quantitative data, numbers we record

• Data collection methods have to be repeatable \(\rightarrow\) we need standardization
• We often identify categories of people or behaviours \(\rightarrow\) classification involves choices
• Data need to be capable of summary \(\rightarrow\) we ignore context

We need to be aware of these limits

We can learn to work ably with quantitative data but that ability must come with awareness of the limits(Nguyen, 2024)

• To make data collection repeatable and observations capable of summary, we have to leave things out, stripping out context, and making classification choices that sometimes reflect biases or blindspots

You are entering a field that has been through a period of revolution

The effects of revolution are present everywhere we do teaching and learning: Simine Vazire calls this the credibility revolution (Vazire, 2018). You will see it in our

• Emphasis on transparency or open science, so our procedures, materials, data and code are open to scrutiny
• Focus on reproducibility, so that our research can be verified
• Commitment to integrity

How we think about and teach Psychological science is different now

• These changes have taken a while (Gelman & Vazire, 2021)
• but now represent the dominant culture in research-intensive universities (like Lancaster)
• These changes mean that critical reflection has a stronger role, now, for staff and students

Meehl’s thinking underlies the reforms in psychological science

Paul Meehls’ writings are interesting and funny. We learn from him that we should

• Be specific in the theory predictions that we test
• Understand that we will learn more when data contradict our predictions
• Than we can learn if observations match our predictions

Doing science and the importance of falsification

Meehl (1990) built on Popper (2005) to argue that we need to be careful about the logic

Doing science and the importance of falsification

Meehl (1990) built on Popper (2005) to argue that we need to be careful about the logic

The logic

Theories and theory-like objects

If only it were that easy

Theories and theory-like objects

• van Rooij (2022) points out that sometimes there are things that we call theories even though they are not theories
• Language is slippery so we need to be careful

Theories and theory-like objects

Auxiliary theories

Meehl (1990) explains that (1.) given a theory (2.) to make predictions we can test (3.) we often need auxiliary theories

We need to be specific in predictions

To get to a prediction we can test then, we need:

• A psychological theory about a process or a thing
• Auxiliary assumptions that gets us from the process or thing to what we can observe
• A specific account of what exactly we are predicting: is the behaviour going to increase or decrease, and by how much?

The theory is not the hypothesis is not the statistical test result

The theory is not the hypothesis is not the statistical test result

Figure 4: Different theories can imply the same process, and different processes can imply the same statistical test result

Given the logic, the challenges: Measurements are imperfect

We cannot look directly inside people’s heads so we must rely on psychological measurements

Given the logic, the challenges: Measurements are imperfect

We cannot look directly inside people’s heads so we must rely on psychological measurements

Given the logic, the challenges: Things vary, people vary

We often assume the same thing happens in the same way in everyone but people vary (Vasishth & Gelman, 2021)

Figure 5: People vary and how conditions or interventions affect us will vary: from strongly negative (dark blue) to null (near-white) to strongly positive (dark red) impacts

Given the logic, the challenges: People are kind of random

Let’s imagine we test a person (you) on a psychological attribute like your vocabulary knowledge.

Given the logic, the challenges: People are kind of random

Now imagine we test two people (you and your neighbour) using the same vocabulary test

Given the logic, the challenges: We usually only test samples of people

Now imagine we test a sample of 29 people. Now we see a distribution of scores

Given the logic, the challenges: Fields of research involve many studies

If we ran the study again, we would draw a different sample of people each time

Figure 12: Six studies, each with its own sample of 20 people

Given the logic, the challenges: Different samples of people will vary

No two samples look exactly alike, even when we’re recruiting people from the same population, and testing everyone with the same procedure

Figure 13: Score distributions from the same six studies

The crud factor

• Meehl (1990)’s crud factor: “everything correlates to some extent with everything else”

What does a hypothesis test show? False positives, false negatives

Because of these challenges, we are often concerned about quality control over our scientific process, and so we seek to limit the rate over a series of studies at which we run into:

• False positives: Concluding we have found something when there is nothing there
• False negatives Concluding there is nothing there when there is something

The hardest science

Science is a thing that is done by people

Our discussion has centered on:

• The logic of the scientific method (Meehl, 1990)
• The ethos or normative principles of modern science (Merton, 1942)
• High-level descriptions of the history of scientific progress (Kuhn, 1970)
• But we need to remember that science is a thing that is done by people

Working with people you don’t know or like or respect or trust

• How is it that groups or communities or networks or societies of people
• Who may not know or like or respect or trust each other
• Can produce this thing: science, a progressively expanding understanding of the world?

Communities in which critical perspectives are necessary and useful

Longino (2026) gives us a good place to start:

• We need free and open discussion of ideas because no-one is perfect and critical discussion should help us to avoid false or partial (personally biased) beliefs (Mill, 1859)
• In practice, this work could progress through the efforts of scientists to test each others’ theories (Popper, 2005)
• As individuals, alone, we cannot get to the truth but we can as a community agree on consensus understandings that will ultimately converge on the truth (Pierce, 1878/1958)

Evaluation of quality, relevance, and interest (novelty or utility)

• How should we evaluate the quality, relevance, interest (novelty or utility) of research?
• In the last few decades, we have mostly relied on peer review
• What is peer review, why is it, and how does it work (or not)?

What is peer review?

Figure 15: The peer-reviewed publication cycle: submission, review, and decision, repeating until accept or reject

Peer review: new, imperfect, maybe coming to an end

• We see peer review emerging over time, in different forms in different places, and for different purposes (Spier, 2002)
• To understand why and how peer review emerges, we can follow the money (Baldwin, 2018):

Scientists and their supporters cast expert refereeing—or “peer review,” as it was increasingly called—as the crucial process that ensured the credibility of science as a whole.

What is peer review?

When you write a research report, you will adopt the structure, the content and the formatting we use when we report our work to our peers (other scientists)

• Introduction
• Methods
• Results
• Discussion

The report is not the work: the hidden realities of lab science

Science is often messy: not all that complexity gets into print

• The things we study (people) are complicated, sometimes difficult to work with, and behave in semi-random ways
• In an ethnographic study of baby labs, Peterson (2016) shows that this means there are sometimes divergences
  • between ideas about how we should do science
  • what we say we did or found
  • what we actually did do

Open science reforms and the evolution of practice

These challenges are why psychologists have come to adopt open and reproducible science practices (Munafò et al., 2017)

• Sharing our predictions, our materials, our data, and the code we use to do analysis
• Improving our methods training: what we are doing here
• Involving multiple labs in collaborative studies (e.g. McMillan et al., 2026: Many babies) to examine variability amongst participants, labs, and methods

Science is a thing that is done by people

Science is a social activity: fads, inequities and biases

• What do we study?
• Who does the studying?
• Fads, inequities and biases

What do we study?

The volume of scientific outputs has grown rapidly over time: growth has been linked to societal changes

Bornmann et al. (2021): Growth in publications over time, split by candidate epochs

What do we study?

Wieczorek et al. (2021) analysed the topics of over 500,000 research article abstracts: different methods and different topics are studied at different times

Wieczorek et al. (2021): We see increasing discourse around specialized, applied and thematically driven topics like addiction, brain imaging or facial recognition

Who does the studying?

Who does what? Does it matter who you are? In a sense, it should not matter.

Merton (1942) says:

The acceptance or rejection of claims … is not to depend on the personal attributes of their protagonist [of who you are, your] race, nationality, class and personal qualities …

Likewise

careers [should be] open to talents [no matter who you are]

Biases and inequities

Who does what? Does it matter who you are? In a sense, it should not matter. But there is plenty of evidence that gender, race, language, geography and gender do matter to:

• What you study
• Your chances of success
• The diversity of the scientific ecosystem of ideas

Biases and inequities: language

• Amano et al. (2023): while most people are non-native English speakers, we face difficulties at every stage

Amano et al. (2023): Estimated disadvantages for non-native English speakers

Biases and inequities: geography

There is a brutal geography at work: whose ideas get heard?

Rodríguez-Pose et al. (2026): For much of the emerging world, the map of frontier research is close to empty

Biases and inequities: geography

There is a brutal geography at work: whose ideas get heard?

• Gomez et al. (2022) analyzed 20 million papers across nearly 35 years and 150 fields:
  • work from outside a global elite is mostly unrecognized
• Zumel Dumlao & Teplitskiy (2025) analyze data from physics journals:
  • if reviewers favor work from their own country (homophily)
  • and most reviewers come from just a few countries
  • then authors from those countries enjoy an advantage

Biases and inequities: gender and ethnicity

Kozlowski et al. (2022) analyzed about 7 million articles to find gender and ethnicity disparities in who does what science

Biases and inequities: gender and ethnicity

• Kozlowski et al. (2022)’s results show:
  • While White authors publish everywhere, a privilege of choice,
  • Minoritized authors tend to publish in scientific disciplines and on research topics that reflect their gendered and racialized social identities
• Unequal representation in science leads to underinvestigation of particular topics and may block innovation

Diversity would be helpful

Hofstra et al. (2020):

historically underrepresented groups often draw relations between ideas and concepts that have been traditionally missed or ignored

Diversity would be helpful, maybe not for those doing the representing

Hofstra et al. (2020) looked at about 1.2 million doctoral dissertations: people from under-represented groups tend to make more distant innovative links between concepts more often but innovative work based on more distant links tends to have less impact

Diversity would be helpful

Figure 16: The people we study vary: we vary. Both sources of variation will have an effect. How we vary shapes the questions that we study, how we study them, and the impacts of the answers we come up with

Diversity would be helpful

Graves et al. (2022) call for socially responsible science grounded in principles of equity

Big ideas and difficult questions

1. Ideas about how we should do science diverge from observations about what we do
2. But we still need to be guided by ideas about communism, universalism, disinterestedness and skepticism
3. We need to be capable of being wrong
4. But getting the logic to work for us is tricky when the thing we study people are varied and kind of random, and the measures we use are imperfect
5. We will get there, and exciting new research shows the way