Week 7: Theory in Science Case Study: Semantic Memory - George S.

Cree

Semantic memory
● A type of long-term memory involving the capacity to recall words, concepts, or
numbers, which is essential for the use and understanding of language
● Tulving (1972) w as the first to define semantic memory

Ross M. Quillian - Quillian was a cognitive psychologist and computer scientist

● His work in the mid-1960s on semantic memory, particularly his PhD thesis in 1966, laid
the groundwork for the development of semantic networks in artificial intelligence

The teachable language comprehender (TLC)

● TLC System
○ In the TLC system:
■ Text Input: The system receives a piece of text, like a sentence from a
textbook, to work with.
■ Processing: It uses its 'brain' (the TLC Program) to think about the text,
trying to understand it.
This 'brain' has some
fixed methods
(permanent routines)
and a special notepad
(temporary workspace)
to scribble notes about
the specific sentence it's
looking at.
■ Memory Check: It then
checks with its
'memory' (the network
memory) to see if what
it thinks it understands fits with what it already knows about the world.
This memory also helps it to create tests or questions to check its
understanding.
■ Asking for Help: If the TLC system isn't sure it has gotten it right, it can
ask for help from a 'smart friend' (an online human monitor), who can
confirm if the understanding is correct or provide more information.
■ Learning: Once the system is sure about the understanding, it decides if
this new piece of information is important enough to remember and then
stores it in its memory for later
Hiearchal Taxanomic Network Theory - Collins & Quillian (1969)

● The cateogry verification task is a

classic experiment in cognitive
psychology that investigates how
people store and access knowledge
●

Hiearchal progressive differentiation in infants

● Frank C. Keil in 1979, pertains to how infants and children develop an understanding of
categories and concepts over time
● As they learn and experience more of the world, they start to differentiate these
categories into more specific ones and understand the hierarchy between them

Warrington & Shallice (1984):

Deficits and Semantics

Diagram (a): Semantic System Division

● Functional Features: Represented

by the green area, indicating a
subsystem of the semantic memory
responsible for processing the
purpose and use of objects
○ Things that are not alive,
understanding the purpose
of things
 ● Sensory Features: Shown in blue, depicting a subsystem responsible for processing
sensory attributes like shape, color, and texture.
○ Things that you can feel, see
● Peripheral Input Systems: These are shown at the bottom, with "Word reps." in red,
indicating the representation of objects through language, and "Visual reps." in brown,
showing visual representation.
● Semantic System: This is where the functional and sensory features converge and
interact.

Diagram (b): Cross-modal Representations

● This part illustrates how different sensory and functional aspects of objects (such as
action, touch, shape, color, sound) are integrated to form a cohesive representation
● Each of these aspects feeds into a central node, suggesting that our understanding of an
object is a composite of its different sensory inputs and associated actions

Category Specific Semantic Deficits

● Inanimate Objects: Patients could provide functional descriptions of these items (e.g.,
"Tent—temporary outhouse, living home.").
● Living Things: Patients' ability to describe these was impaired (e.g., "Parrot—don't
know.").
● This contrast highlights how damage to specific semantic subsystems affects the
ability to categorize and describe different types of objects, consistent with the
Sensory/Functional Theory

Brain scans

● There are two scans shown, likely indicating areas of the brain affected by damage that
leads to semantic deficits. These scans
support the idea that different brain
areas may be specialized for different
types of semantic processing

Double dissociation (Hillis and Caramazza 1991)

● At 6 months post-onset, patient JJ shows

relatively good naming performance for
 land animals, water animals, birds, and furniture but poor performance for vegetables,
fruit, and other food items. In contrast, patient PS shows poor performance for naming
animals but better performance for vegetables, fruit, and furniture.
● At 13 months post-onset, both patients show changes in their naming abilities, but the
dissociation between the categories remains. JJ still shows a clear deficit in naming
vegetables and fruit, while PS continues to have difficulty with animals.
○ JJ has a deficit in accessing the semantic information for certain food items but
not for animals
○ PS has a deficit in accessing the semantic information for animals but not for most
food items

The term "double dissociation" comes into play when two patients have lesions in different parts
of the brain, and each patient's lesion leads to deficits in different cognitive functions. This helps
to demonstrate that the two cognitive functions are controlled by different regions of the brain

● This pattern suggests that the cognitive processes underlying the naming of animals and
the naming of certain foods are independent of each other and likely localized in different
brain areas

How we interpret the world can be

categorized into three systems
1. Orthographic
a. The way you read it as
represented by letters
2. Structural/visual representations
a. The way you see the
object visually
3. Phonological input
a. The way you pronounce
the word
Domain Specific Hypothesis: Caramazza and Mahon (2003)

● This hypothesis suggests that different categories of objects are processed differently
in the brain, and certain neural circuits are specialized for processing certain types
of objects
○ The domain-specific hypothesis is important because it helps us understand how
information is organized in the brain
○ It suggests that there might be different regions of the brain that specialize in
processing different kinds of objects

Extended Sensory/Functional Theory

● The Extended Sensory/Functional
Theory posits that different
regions of the brain are
specialized for processing
information about objects based
on sensory experience (like how
an object looks) and functional
experience (like what an object is
used for)
 ● This theory extends beyond the simple sensory/functional distinction
● There's a network of regions involved in integrating this information to form our
understanding and categorization of different objects

Problems with Hierarchical Taxonomic Theory

➔ By the late 1970’s, a number of problems had emerged in the literature
➔ What should people be able to recognize more?
◆ A cow is an animal
◆ A cow is a mammal
➔ The theory doesnt really make sense

Problems with Category Verification Task

➔ What variables would you need to control/manipulate in order to trust data from a
category verification task experiment?
◆ Proportion of True vs. False statements
● How many of the statements that participants are asked to verify are
actually true, and how many are false.
◆ Number of categories used in the experiment
● The total number of distinct categories that the concepts in the experiment
belong to.
◆ Number of items per category
● How many different items or instances are included within each category.
◆ Word frequency
● How often the words representing the concepts appear in the language that
participants speak.
◆ Typicality of concept in category
● How representative the concept is of the category to which it's supposed to
belong.
◆ Familiarity of concept and category
● How well-known or common the concept and its category are to the
participants.
◆ Age of Acquisition of concept
 ● The average age at which people generally learn the concept.
◆ Etc.
● This could include additional factors like the visual complexity of any
images used, the abstractness of the concepts, the semantic distance
between concepts and categories, or the modality of presentation (visual,
auditory, etc.).

Spreading activation theory

➔ People organize their knowledge of the world based on their personal experiences, which
in turn form the network of ideas that is the person's knowledge of the world

Problems with SAT

➔ Cataloguing Nodes and Connections:

◆ List and define all concepts
(nodes).
◆ Identify and describe all
connections (edges) between
concepts.
➔ Determining Strengths of Weights
Between Concept Nodes:
◆ Analyze data on frequency and
context of concept usage.
◆ Calculate statistical correlations
between concepts.
➔ Is it Falsifiable?
◆ Design experiments to test the
predictions of the network model.
◆ Ensure the model can be challenged by empirical evidence

Factors Underlying Semantic Structure Cree & McRae (2003)

Week 8: Neural networksand semantic cognition

Approaches to modeling cognition and the brain

● Verbal models
● Formal mathematical models
● Computer programs that manipulate symbols
● Bayesian models
● Neural network models

Neural Network Models of Cognition

● Can neural network models learn to do things humans do? Yes! The brain is, after all, just
a big, wet, neural network
● How does the brain figure out how to wire up the neurons so that we can see, speak, act,
etc.
○ By studying simpler neural systems, such as neural network models, we may be
able to figure out how the brain does it

Main building blocks

Donald Hebb - 1949

➔ Hebb's rule, a theory in neuroscience that proposes an explanation for the adaptation of
neurons in the brain during the learning process, described as "cells that fire together,
wire together." This means that the connections between neurons strengthen as they
are activated simultaneously
➔ Cells that fire together, wire together

Frank Rosenblatt - The Perceptron 1950

➔ The perceptron is an early neural network model
invented by Frank Rosenblatt
 ➔ A linear classifier, meaning it makes its decisions by weighing input signals, summing
them, and passing them through a step function to produce an output

Learning distributed representations of concepts - Geoff Hinton 1986

● Concepts (like words or ideas) can be represented across different neurons rather than
being localized to a single neuron
● Each concept (agent, relation, patient) is distributed across multiple neurons, which
allows for a more flexible and robust representation of information

Neural Network Models of Semantic Memory - David

Rumelhart (late 1980s early 1990s)
● The model uses layers to represent different types of
information (items, attributes, and relations), and how
these are processed to form an understanding of
semantics (meaning)
Layers of Units

Net Input

Activation

●
A little history…
➔ 1974: Back-Propagation was invented by a Harvard PhD student
◆ Neural network: feeds forward information
◆ Is an algorithm to test errors which will travel back from input nodes to output
nodes
● Applied to reduce cost function and errors
● The previous layer output is multiplied with weight which gives the
activation function (forward propogation)
● Output highest activation = suitable output marth for the corresponing
input
● BP is the tool to calculate the gradient of the lost function
○ Quick, easy, no extra functions
◆ Based on biological neural network
◆ Neurons = nodes
➔ 1980’s: rediscovered in the 1980’s by David Rumelhart and David Parker
➔ 1986: First popularized by Rumelhart, Williams, & Hinton (1986)
◆ Currently used to train neural nets in many different applications within
psychology and beyond (e.g., pre-cancerous cell detection, predict exchange rates,
adjust telescope optics, etc.)

The 4-2-4 problem

● The network's job is to learn a way to turn the input into the correct output
● Each pattern consists of a sequence of 4 inputs and the corresponding 4 outputs
● The 4-2-4 problem tests the network's ability to compress and then reconstruct
the input data
Week 10: AI and Psychology

What made deep learning possible?

➔ A new activation function, replacing the sigmoidal function with the Rectified Linear
Unit (ReLU)
➔ Training on graphics cards (as opposed to CPU)
➔ Access to lots and lots of data

Large Language Models

➔ Massive statistical models that predict the next word, phrase, sentence, or paragraph,
given an input
➔ Given the input: What is the capital of France?
◆ It can produce output such as: The capital of France is Paris
◆ How does the AI model know how to generate this type of information?
Word meaning depends on context
➔ Simple word vectors like these dont capture the fact that words have multiple meanings
➔ LLMs like ChatGPT can represent the same word with multiple vectors to capture this
information
➔ Our natural language is full of many ambiguities
➔ People use context to resolve these ambiguities
◆ Word vectors provide a flexible way for LLMs to represent precise meaning in a
particular passage of text

Training set
Sentences were formed from
- Relative pronouns
- Who
- Nouns
- Boy, girl, cat, dog, boys,
girls, cats, dogs
- Proper nouns
- John
- Mary
- Verbs
- Chase
- Feed
- See
- Hear
- Walk
- Live
- Chases
 - Feeds
- Sees
- Hears
- Walks
Sentences were created from these words folling a grammar check
- John feeds dogs
- Girls feed dogs

Conclusion
- The network developed hidden unit representations that can distinguish between the
occurrence of the same word (e.g., boy) in different grammatical positions in a sentence
(e.g., subject, object), and use that information to correctly predict the next word in the
sentence
- Elman interpreted this as evidence that neural networks could code for the complex
structural relationships inherent to language, and could learn this from mere exposure to
the language
- His model was just a toy example. It would take another 30 years for people to figure out
how to build more powerful systems that could do the same with real, natural language

What made deep learning possible?

● The parts of words are stored in the model as tokens. Each token is represented as a
vector of numbers
○ The vectors work similarly to map coordinates. Paris, for example, is at [2.34337,
48.85663]
○ Vectors tell us about the relationships between words
● The vectors tell us about the relationship between the words (places in the context of our
example)
● The vectors assigned to the word tokens in the model similarly place them in a “space”
and similarly encode how “close together” they are in that space
● The distances are not expressed in 2 dimensions or 3 dimensions
● The distances are represented in hundreds of dimensions. Each number in this vector us
the value on a single dimension
○ We dont know what each number codes but it may be something like
substitutability
● When you type a prompt on your phone, the model is trying to generate the next token in
one dimension, the model is trying to predic the next token (word) in response to the
prompt
What are LLMs actually doing?
- They are NOT simulating human intelligence
- They dont have the non language specific cognitive capacities required for modeling
thought
- They ARE predicting the next phrase or sentence, given what they have been exposed to
in the training data
- The output can be thought of as the “average” of what the internet and popular books say

How are LLMs Evaluated?

- According to its creators; chatgpt can pass the bar exam, graduate record exam (GRE),
and score well on advanced placement tests
- What are the standards that constitute good LLM performance?
- Computer scientists: tends to output text that is grammatical, coherent, and natural
sounding
- Psychologists

What does the field need to advance?

- Keystone data sets: large, natural language databases that meet the highest standards for
expert and impact evaluation
- Must represent populations and constructs of interest
- Must rely on precise definitions of these constructs
- Should have expert annotations
- Must be linked to psychologically important outcomes
- Real world behavior, mindset, mental health outcomes etc.
How will AI transform psychology?

➔ Therapeautic chatbots
➔ Tools that automate note taking and administrative tasks
➔ More intelligent training and intervention

How will AI transform psychology?

Research

● AI for providing new forms of data collection (e.g., monitor social media activity and
link to physiological metrics).
● AI as the participant.
 ● AI for writing code, including data analysis, data visualization, and more complex
projects.
● AI for writing and editing articles.
● AI for reproducing and extending analyses (e.g., repeat a meta-analysis that was
conducted 10 years ago including all the newly published data).
● AI for reviewing articles

Biases

● Because generative Large Language Models (LLMs) are trained on text produced by real
humans, they embody the same biases held by those humans, including gender, race, and
cultural biases.
● This has already led to discrimination due to the uncritical application of algorithms to
new tasks, such as filtering job applicant files.
● Significant, serious thought must be given to new applications in terms of what biases
could exist in the model due to the training set, and how things could go wrong if the
model is applied uncritically to new tasks.
● Most current AIs are created and owned by corporations, whose main motives are profit.
The motivations and goals of scientists, especially psychologists, are quite different,
involving stricter moral guidelines, including a duty to care for participants and clients
Week 11: Meta-Analysis

What is Meta Analysis? (The statistical synthesis of the results of different studies
addressing similar questions)
● First used in 1970
● Part of a broader field of research synthesis techniques that includes systematic reviews

Why do a meta analysis?

1. To resolve discrepancies in the outcomes of a group of studies
2. To highlight where more research is needed or where evidence is already sufficient
3. To avoid biases, providing replicability and transparency in reviewing research evidence
4. To allow broader generalization by encompassing a broader scope then is possible in any
one study

How big should your meta analysis be?

● Meta-analyses can range from small and narrowly focused to very large, broad in scope,
and general. How large should yours be?

The goldilocks (just-right-size) meta analysis rule should depend on

1. the questions you are asking of the literature
2. the size of the literature
3. the scientific discipline in which you are working
4. your resources for conducting the meta-analysis

Introduction - size and scope

How do you know if you’ve chosen the right number of studies and the right scope (boundaries)
for your meta-analysis?

Decide which review youre gonna do

1. Scoping review - more narrative, map existing literature and find gaps
a. Qualitative
2. Systematic review - thorough and unbiased search, then assessment
a. Quantitative

Systematic reviews (a methodology for collecting, evaluating, and reporting of the data that go
into a meta-analysis (contrast with traditional narrative reviews). Consists of search, then
assessment
 ● Involves an a priori literature search strategy with replicable search terms and criteria for
selection of studies, reported in a protocol following standardized guidelines (e.g.,
PRISMA)
● Involves searches are conducted online using multiple scientific literature databases (e.g.,
Web of Science, Scopus, PsycInfo). Unpublished data may be included
● Involves search and screening should be done by at least 2 people

PRISMA
● Preferred Reporting Items for Systematic Reviews and Meta-Analyses
● Evidence-based minimum set of guidelines aimed at helping authors improve the
reporting of systematic reviews and meta-analyses
● PRISMA focuses on ways in which authors can ensure the transparent and complete
reporting of this type of research
● The guidelines include a 27-item checklist and a four-phase flow diagram to assist in the
systematic reporting of reviews and meta-analyses related to healthcare interventions,
though the principles can be applied more broadly to other fields as well. The aim is to
enhance the clarity, transparency, and rigor in the reporting of systematic reviews and
meta-analyses, facilitating the critical appraisal and replication of such studies

Meta Analysis Effect Sizes: We use effect sizes to summarize the findings that were reported
using different measures
● Once the systematic review of the literature is complete, we can start the meta-analysis
● First, we need to put the results of the different studies on the same scale
○ Analogy: cash gifts at wedding in different currencies - who gifted the most?

Meta Analysis Effect Sizes: Which effect size metric should you use?
● One common effect size metric is the standardized mean difference
● ​This is the difference between the means of two groups, divided by their pooled standard
deviation
● This is relatively easy to interpret: the effect size indicates how many standard deviations
the target group scores better (or worse) than the comparison group. 0.0 indicates no
difference
● Other alternatives include
○ Correlation coefficients
○ Log response ratio
○ Odds ratio
○ Hazard ratio

Meta Analysis Effect Sizes: How do you get these values?

 ● Correlation coefficients and odds ratios are commonly reported, so you just lift them
from the paper
● Standardized mean differences between groups and response ratios may need to be
calculated from the information given, sometimes lifting it from graphs (using, e.g.,
ImageJ), or from scouring online supplements
● It is becoming more common for researchers to report effect sizes

Meta Analysis Effect Sizes: What do you do if the results are reported in the opposite direction?
● Some studies may report results in the opposite direction of the results of the other
studies (e.g., most people study vitality, but some report fatigue).
● What should you do?
○ Reverse-score so that the higher scores indicate comparable responses, allowing
them to be meaningfully combined
● Overall, be clear about what you are measuring, and align this with the question(s) you
are asking

Meta Analysis Effect Sizes: Code for moderators at the same time
● Moderator: a third variable that influences the relationship between two other variables
(e.g., mental health may moderate the relationship between sleep quality and academic
achievement)
○ Examples include participant ages, gender, income, level of education, etc.
● As you are going through the studies, getting the effect sizes, note down characteristics of
the study that vary across studies and could potentially influence results

Aggregating Effect Sizes Across Studies (Basic premise: effect size from each study is
combined, weighted by sample size)
● Larger studies are assumed to be a better estimate of the “true effect”
● Why? Small studies are more susceptible to chance outcomes
● Therefore, we give larger studies more weight when we combine the results

Aggregating Effect Sizes Across Studies: Larger studies are a better estimate of the “true” effect
● If we know the statistical properties of an effect size metric, we can calculate the
sampling variance (the variance in that effect size if we were to do the same study many
times)
● This is what we use to weigh the larger studies more heavily
● The logic is that every study is assumed to estimate some “true” outcome. If the sample
size was large enough, we would get close to that true effect

Aggregating Effect Sizes Across Studies: Fixed vs. Random Effects Models
 ● If we assume that the only cause of variance in the effect sizes across studies is the
sampling variance, then we are using a
○ fixed effects model
● But in reality, there are likely to be more sources of variance than just the sampling
variance (e.g., location). If we factor these in, then we are using a
○ random effects model
● Fixed effects models are more like to generate false positives. They were popular when
meta-analysis first hit the scene, but random effects models are more popular today

Aggregating Effect Sizes Across Studies: How do we get the “results”?

● We use equations to average the weighted effect sizes across the studies
● The results are usually reported as a mean effect size, with its 95% confidence interval
● Heterogeneity statistics and p-values indicate whether the mean effect differs from the
value for “no effect”
● The grand mean effect size is said to be statistically significant if its confidence interval
does not cross 0 (or 1 if using odds ratios or relative risk)
● These results are almost always presented as a forest plot

Examning Heterogeneity
● Heterogeneity: the variability in the true effect across studies
● Imagine that we could run the same study multiple times, in the same way, with the same
experimental protocols, same participants, etc. Any differences in results would be due to
 measurement error (difference between measured value of a quantity and its unknown
true value), and the studies would be homogeneous
● Heterogeneity denotes the variability in outcomes that goes beyond measurement error
alone

Examning Heterogeneity
● Heterogeneity: the way a study differs from the others, such as methodology,
characteristics of population, specifics of the intervention, etc. We lump these all together
into the heterogeneity term
● More heterogeneity means the mean effect size is less likely to reliably characterize
responses across studies, and therefore, may not be useful for providing sound
recommendations for practice or policy
● If there is a great deal of heterogeneity among studies, we interpret that to mean that the
true effect (the real response to the intervention) differs among studies

Indices of Heterogeneity
● Indices of heterogeneity help specify whether variation in the mean effect across studies
is reliably different from 0
● Example
○ Q-test: used to determine whether excess variation is present by change alone
○ I2: used to estimate the proportion of the total variation in true effect sizes that is
due to heterogeneity in the real responses across studies
■ 25 = low, 50 = medium, 75 = high heterogeneity

Why does it matter if there is high heterogeneity?

● Why does it matter? Some of the heterogeneity may not be due to random variation, but
instead, to factors that can be identified. We call these moderators
● Examples of common moderators include age, education level, duration of each study,
and environment in which the study was conducted
● Remember the extra factors we coded for when getting the effect sizes? We can use these
in moderator analyses to better understand the phenomenon under study

Study heterogeneity: Q-statistic

● Q-Statistic: are all of our studies capturing the exact same effect? Or are all of our studies
a little bit different, not capturing the exact same effect?
● It is a NHST that evaluates the null hypothesis that all studies are examining the same
effect. Therefore, a statistically significant Q indicates that the included studies do
not capture the same effect
● Note that we can’t conclude anything from a non-significant Q
 ○ Note also that Q is prone to underestimate heterogeneity in small samples and
overestimate in large samples

Study hetergoneity: I2
● I2 is a percentage that represents the proportion of observed variation that can be
attributed to the actual difference between studies (such as age of participants in different
studies) rather than within-study variance (differences between the individual participants
in a single study)
○ 25 = low; 50 = medium; 75 = high variance
● Not sensitive to the number of studies included

Study hetergoneity: Tau Squared

● Tau-squared can also be used to estimate the total amount of study heterogeneity in
random-effects models
● When Tau-squared = 0 this means no heterogeneity
● Theoretically, tau-squared can range from 0 to infinity

Study hetergoneity: Baujat plot

What does this mean?

Y axis: combined effect sizes
X axis: heterogeneity
 - Studies located further to the right on the X-axis: These are contributing more to
the heterogeneity. High values here indicate that a study differs more
significantly in its effects compared to other studies in the meta-analysis.
- Studies located higher on the Y-axis: These have a greater influence on the
overall effect size estimate. A study positioned higher on this axis would
significantly alter the combined effect size if it were removed from the analysis
- Baujat plot is a valuable diagnostic tool in meta-analytical reviews,
detecting outliers and assessing heterogeneity

Publication bias: funnel plot

● Publication bias happens when studies with
non-significant results are less likely to be
published
● If a significant positive effect were found for the
summary effect size, the vertical line should be to
the right of zero (it shows where the summary
effect size is from the forest plot)
● A non-significant effect would be close to zero
● If the funnel is uneven, with more points to the
right of the line, this is evidence for publication
bias