Experimental

Design
Reading Report
Reading reports from scientific papers, book chapters and
other materials from the Experimental Design Class 2023
from Professor Diego Pinto.
By Cristiane Lell 20221723, Nova IMS, December 2023.

1
Table of Contents
Book Chapters

Chapter 1 - Experiments and Generalized Causal Inference

Chapter 2 - Statistical Conclusion Validity and Internal Validity

Harvard Business Review

1. How to design smart business experiments.

2. How to Design (and analyze) a Business Experiment
3. A Refresher on Randomized Controlled Experiments

Exemplary Research

1. Consumer goal pursuit, By Juliano Laran,2016.

2. Automatic Effects of Brand Exposure on Motivated Behavior: How Apple Makes You “Think Different, By
Grainne M. Fitzsimons, Tanya L. Chartrand, Gavan J. Fitzsimons,2017.
3. From Fan to Fat? Vicarious Losing Increases Unhealthy Eating, but Self-Affirmation Is an Effective Remedy, By
Yann Cornil and Pierre Chandon,2013
4. Going Green to Be Seen: Status, Reputation, and Conspicuous Conservation, By Vladas Griskevicius, Joshua
M. Tybur, Bram Van den Bergh,2010.

Chapter 1
Experiments and Generalized Causal Inference

2
In the exploration of experiments and their role in scientific inquiry, this first chapter delves into the historical
evolution of experimentation. A notable shift from passive observation to active manipulation occurred in the
17th century, marking a pivotal moment in the advancement of scientific understanding. This transition
allowed researchers to discover the regularities and possibilities of the world through deliberate actions
followed by systematic observation.

Control of extraneous influences emerged as a crucial consideration, initially developed for natural sciences
like astronomy and physics. However, applying these controls in fields such as public health proved
challenging due to the complexity of external influences. Consequently, new methods like random assignment
were introduced to manage extraneous factors and enhance experimental designs. Over time, experiments
evolved, accumulating theoretical and observational experience across diverse fields. Researchers identified
biases and developed methods to address them, contributing to the sophistication of experimental designs.
The essence of experiments, whether in scientific or everyday contexts, revolves around deliberately varying
an independent variable to observe its effects on a dependent variable.

Cause, Effect, Causal Relationships: To better understand experiments, it is essential to learn both a
vocabulary for discussing causation and the key underlying concepts. Defining cause, effect, and causal
relationships is a complex task. Causal relationships are often intuitively recognized in daily life, such as
attributing car damage to another vehicle's impact or relating test grades to study hours. However, a precise
definition of cause and effect has challenged philosophers for centuries.

Þ Examining the idea of "cause," think about a forest fire: various factors contribute to it, forming what Mackie
called "inus conditions." Individually, these factors aren't enough, but together, they create a unique set.
Understanding this complexity is key for grasping probabilistic relationships.
Þ The notion of "effect" is explained through a counterfactual model rooted in David Hume's philosophy. An effect
is the difference between what happened and what would occur without treatment. Creating a high-quality but
imperfect source of counterfactual inference in experimental design is crucial, acknowledging the qualitative
nature of causal inference.
Þ Causal relationships are established when the cause precedes the effect, the cause is related to the effect, and
no plausible alternative explanation exists. This classic analysis, formalized by John Stuart Mill, aligns with
experimental methods where the presumed cause is manipulated, and outcomes are observed, ensuring
temporal precedence, and examining variations in cause-effect relationships.
Þ Causation X Correlation: When considering causation versus correlation, the maxim "Correlation does not prove
causation" emphasizes the need for in-depth research. Correlations alone can't prove cause-and-effect; third
factors (confounds) may influence both variables. Recognizing and understanding confounds is vital for
discerning genuine cause-and-effect connections in research.

Manipulative and Nonmanipulative Causes: In experiments, causes under investigation must be

manipulable, allowing researchers to alter them intentionally and observe the resulting effects. However,
nonmanipulable events or attributes cannot be causes in experiments because they cannot be deliberately
varied. While not all causes are directly manipulable, experiments primarily focus on manipulable causes due
to their investigatory advantages. For instance, a genetic defect causing a condition like PKU is
nonmanipulable, yet its impact can be explored indirectly in nonexperimental studies or by manipulating
processes that inhibit its effects, like using a diet to mitigate its biological consequences. Studying

3
manipulable agents, like diets, offers advantages in providing direct solutions and improving the quality of
counterfactual inference, particularly through methods such as random assignment.

Nonetheless, nonmanipulable causes should be studied using available means because understanding them
can eventually lead to the discovery of manipulable agents that can address the problem. Analogue
experiments or natural experiments may sometimes be conducted on nonmanipulable causes, though these
methods may not fully replicate the reality of the cause being studied. Despite the advantages, experiments
are acknowledged as imperfect tools for investigating causes. Some causes are inherently nonmanipulable,
and experiments may struggle to fully replicate real-life experiences. The complexities of causation emphasize
the need for researchers to critically evaluate and design studies, considering the strengths and limitations of
different research methods.

Causal description and Causal Explanation: Causal descriptions focus on the "what" and "how" of cause-
and-effect relationships, while causal explanations delve into the "why" and "how." Experiments excel at
describing the effects resulting from intentional treatment variations, termed causal description, but they
often fall short in explaining the mechanisms and conditions underlying these causal relationships, known as
causal explanation. While individuals may easily grasp causal descriptions—such as flicking a light switch
leading to illumination—explaining why these actions cause certain effects involves dissecting treatments into
essential and nonessential features, understanding causal efficacy, and identifying mediating processes.

Causal explanation becomes crucial when treatments fail to produce expected effects. Understanding the
underlying causes aids in troubleshooting, directing attention to resolving issues—like detecting and fixing a
short circuit if a light fails to illuminate. Moreover, causal explanation helps generalize causal descriptions,
identifying essential features applicable in varied situations. The pursuit of explanatory knowledge becomes
pivotal post-discovery of novel causal relationships, leading scientific efforts toward understanding the how
and why behind these findings. However, while experiments might not always provide extensive explanatory
causal knowledge, they serve essential roles in scientific practice. They validate chains of causal links, test
competing explanatory theories, examine varying effects under different conditions, and contribute
observations to understand the mediating variables within the causal chain.

Despite experiments leaning towards improving causal descriptions rather than explanations, their focus on
molar events—those less directly linked to outcomes compared to molecular mediating processes—still yields
dependable causal relationships. Statements like school desegregation causing white flight or psychotherapy
enhancing mental health remain valuable in policymaking, practice, and scientific inquiry despite their
descriptive nature.
MODERN DESCRIPTIONS OF EXPERIMENTS

Modern experimentation involves various terms, some precisely defined and consistently applied, while
others are more ambiguous. The central theme across all experiments is the control of treatment, which can
manifest in diverse forms. Mosteller (1990) emphasizes that in experiments, investigators manage the
application of the treatment, and Yaremko et al. (1986) define experiments as studies where one or more
independent variables are manipulated to observe effects on dependent variables. The authors define them
as: Randomized Experiment: Credits Sir Ronald Fisher, employs deliberate interventions with random
treatment assignment, considered a gold standard in fields like medicine. Quasi-Experiment: Introduced by

4
Campbell and Stanley, tests causal hypotheses without random assignment, relying on self or administrator
selection. Natural Experiment: Examines naturally occurring contrasts, especially in economics, when
randomization is impractical. Nonexperimental Designs: Encompassing correlational, observational, and
nonexperimental designs, lack random assignment, with critics questioning their ability to support strong
causal inferences.
EXPERIMENTS AND THE GENERALIZATION OF CAUSAL CONNECTIONS

Experiments offer a unique way to understand causal relationships, but their strength lies in local contexts,
posing challenges for generalization. Most experiments are conducted in specific settings, using certain
versions of treatments and limited measures, making their results seemingly confined. However, the aim often
extends beyond the immediate study to connect findings with broader theories or policies, seeking
generalization at the level of concepts and across diverse populations and settings.

Most Experiments Are Highly Local but Have General Aspirations: Experimentation's power lies in
elucidating causal inference, yet its vulnerability surfaces in uncertainty regarding the broader applicability of
the established causal relationship. A distinctive aspect of this book is its emphasis on generalization, aiming
to address and navigate this inherent challenge. The challenge of generalization in experiments lies in their
inherent localized nature. Most experiments are conducted in specific settings with particular versions of
treatments, using a convenient sample of people at a specific point in time. The measures employed often
differ theoretically, and the experimental context becomes a historical reference. However, the significance of
experimental results is typically not confined to the specifics of the local study. Researchers and policymakers
aim for broader implications, seeking to connect results to theoretical constructs and policies with wider
applicability. Theorists focus on generalization at the linguistic level of constructs, while policymakers are
interested in the probabilistic extension of causal relationships across diverse sites.

Despite the conflict between localized causal knowledge and broader causal goals, experiments
demonstrating limited generalization can be as valuable as those with broad implications, for instance,
Physicists may discover elements that aren't meant for real-world application, and social scientists might
uncover effects under specific conditions, not expecting widespread occurrence. Such instances can validate
or refine theories without aiming for broader implications. The theory of causal generalization presented in
the subsequent chapters integrates Cronbach's ideas, addressing both construct validity generalizations and
external validity generalizations concerning variation in persons, settings, treatments, and measurement
variables.

Construct Validity and Causal Generalization as Representation: The challenge of construct validity in
causal generalization involves transcending specific instances observed in an experiment to abstract
constructs they represent. These constructs are often more conceptual than the particular instances involved.
They might refer to individual elements or the relationships among them, including causal links. Consider an
experiment studying patient education's impact on recovery after surgery. The study's specifics, like the
educational course given to patients, the outcome measures used, and the hospital setting, need to be
connected to broader constructs: patient education (cause), physical recovery (effect), surgical patients (units),
and hospital settings (context).

5
Researchers often confront empirical data that demand reevaluation of the study domain. This might lead to
a more refined inference, narrowing down the initial understanding. For instance, if the informational
component of patient education proves causally related to recovery but the hospital tour does not, the causal
agent might need to be redefined. Conversely, data might prompt researchers to think in broader terms,
framing treatments as subclasses of broader interventions or recovery as a subclass of larger categories like
"personal coping."Over time, there's a dynamic interplay between the researcher's initial categories, the actual
study, its outcomes, and subsequent interpretations. This ongoing process might alter the researcher's
conceptual understanding based on the achieved study particulars and reader feedback. Yet, at its core, the
challenge remains: How can we extrapolate from observed instances and associated data patterns to the
specific target constructs they represent?

External Validity and Causal Generalization as Extrapolation: The second challenge in generalization
involves determining whether a causal relationship holds across variations in individuals, settings, treatments,
and outcomes. For instance, consider someone reviewing the results of an experiment on the impact of a
kindergarten Head Start program on reading test scores for poor African American children in Memphis. They
might wonder if a program with similar goals would be as effective in enhancing math test scores for poor
Hispanic children in Dallas if implemented in the present.

This example highlights that generalization doesn't imply broader application universally. It involves
extending insights from one context or population to another. Generalization can be from narrow to broad,
as seen when researchers randomly sample participants from a national population to make probabilistic
inferences about the entire population. Conversely, it can go from broad to narrow, as exemplified when
policymakers, considering the continuation of a program like Head Start, focus on national averages rather
than specific local outcomes.Whether it's moving from narrow to broad, broad to narrow, or across units at a
similar level of aggregation, all these instances of external validity questions share a common goal: inferring
the extent to which a causal effect holds under variations in persons, settings, treatments, or outcomes.

Approaches to Making Causal Generalizations: Causal generalization, or the extension of causal

relationships beyond specific instances, poses a challenge for experiments. Experiments, like other research
methods, often involve a limited set of operations, making them highly localized. This limitation is shared with
case studies, performance monitoring systems, and marketing questionnaires. The challenge is justifiable
belief in an experiment's ability to bridge the gap between specific sampling details and more general
inferences.

Sampling and Causal Generalization: Sampling methods emerge as crucial tools for addressing these
generalization challenges. Formal probability sampling, purposive sampling of heterogeneous instances, and
purposive sampling of typical instances each play a role in bridging the gap between specific instances and
their target constructs. However, these methods are not without limitations, particularly in defining target
populations and achieving comprehensive samples that represent all variations. Recognizing the limitations
of sampling methods, a grounded theory of causal generalization is presented. This theory, grounded in actual
scientific practices, outlines five interconnected principles: Surface Similarity, Ruling Out Irrelevancies, Making
Discriminations, Interpolation and Extrapolation, and Causal Explanation. These principles guide scientists in

6
making causal inferences by assessing similarities, identifying relevant aspects, clarifying discriminations,
exploring inferences within and beyond sampled ranges, and developing and testing explanatory theories.

A Grounded Theory of Causal Generalization: The book introduces a grounded theory of causal
generalization, originating from construct and external validity literature. This theory outlines how scientists
make generalized conclusions about causal connections through five closely interlinked principles. These
principles guide scientists in assessing similarities, ruling out irrelevancies, making discriminations, conducting
interpolations and extrapolations, and developing causal explanations. The theory acknowledges analogues
in construct validity literature and cognitive science, emphasizing the application of these principles in
scientific theory and practice. Throughout their research, scientists consistently apply these principles, from
reading and designing studies to data analysis and literature reviews. These principles are not confined to
individual studies but extend to considering how their work fits into a broader body of research. While formal
probability sampling is recommended when feasible, the authors stress that it is just one method among
many that scientists use for causal generalization, which includes practical logic, statistical methods, and other
design features.

EXPERIMENTS AND METASCIENCE

Experimentation, a practice as old as humanity, predates philosophical debates about causation and
generalization by thousands of years. Despite extensive philosophical discussions on experimentation, its role
and significance in science have remained constant over the past 400 years. Experimentation, described by
Hacking as having "a life of its own," stands as one of science's most powerful methods for uncovering causal
relationships. Scientists, although not required to delve into sophisticated philosophical reasoning, can
benefit from understanding these debates. While justifying the practice of experimentation today doesn't
necessarily demand intricate philosophical arguments, it aids scientists in gaining insights. Distinctions such
as molar and molecular causation, descriptive and explanatory cause, and probabilistic and deterministic
causal inferences provide a framework for both philosophers and scientists to comprehend the purpose and
results of experiments. Beyond philosophical considerations, critiques from the history, sociology, and
psychology of science contribute to a broader understanding of the nature of experimentation. Some of these
critiques explore the justified role of experimentation and highlight its limits in both science and society.

Science and Trust: Philosopher Kuhn described scientific revolutions as shifting paradigms, influencing
thinkers like Popper, Toulmin, and Feyerabend. These critiques challenge the certainty of scientific knowledge,
suggesting that experiments provide only ambiguous results due to underlying assumptions and biases. This
skepticism extends to modern sociological critiques, revealing the influence of social, psychological, and
political factors on scientific knowledge.

Trust plays a significant role in science despite its skeptical image. Scientists rely on established methods,
findings, and theories—trusting them more than doubting. However, these criticisms highlight the limits and
uncertainties in scientific knowledge. Experiments, considered windows to nature, offer fallible knowledge
shaped by theoretical assumptions and contexts. Scientists often identify as ontological realists but weak
epistemological relativists, acknowledging the role of external factors in shaping scientific theories.

These critiques emphasize the limitations of experiments in revealing absolute truths. They shed light on the
human elements within scientific endeavors, acknowledging biases, errors, and social influences affecting

7
outcomes. While experiments remain crucial, they aren't infallible, and their interpretations are influenced by
various factors. Stubborn facts emerge, offering dependable results, but experimental evidence isn't always
overshadowed by theories, as exploratory experiments often drive scientific progress. Despite limitations,
experiments provide a substantial factual base, contributing significantly to scientific knowledge over time.

A World Without Experiments or Causes: In a hypothetical scenario where our knowledge base is erased,
the idea of manipulable causes and experiments would likely re-emerge due to their practical usefulness in
understanding and thriving in the world. The concept of manipulable causes holds value for survival, leading
us to explore and refine these ideas through experimentation. Over time, the refinement of experimental
methods would involve grappling with issues of causation, counterfactual inference, and alternative
explanations. This ongoing process of experimentation and refinement aims to enhance the quality of causal
inferences, especially in complex field settings, and improve our ability to generalize these inferences across
various factors like persons, settings, treatments, and outcomes.

Conclusion

In summary, this exploration of experimental and quasi-experimental designs for generalized causal inference
uncovers a dynamic journey in understanding causation. From the historical evolution of experimentation to
the complexities of causation, the text emphasizes the importance of deliberate manipulation, systematic
observation, and the recognition of nuanced relationships. Key takeaways include the distinction between
causation and correlation, the challenges of manipulable causes, and the dual role of experiments in both
describing and explaining causal relationships. Despite their limitations, experiments serve as valuable tools,
similar to building blocks, contributing to our understanding of the intricate web of cause and effect. The
report underscores the need for critical evaluation, recognizing the localized nature of experiments while
aspiring for broader implications. In the broader context, experiments, despite skepticism, remain integral to
scientific progress, offering fallible yet substantial contributions to our evolving knowledge of the world.

Chapter 2
Statistical Conclusion Validity and Internal Validity

8
The study of Clever Hans, a math-solving horse1, revealed researcher cues influenced its actions, challenging
the validity of the initial inference. This chapter introduces the validity theory underlying generalized causal
inference, exploring its meaning and presenting a typology covering validity types and threats. It's all about
ensuring the truthfulness of inferences drawn from empirical evidence, not just about methods or designs.
Validity refers to the approximate truth of an inference based on evidence, including empirical findings and
consistency with other knowledge sources. In Campbel words:

“We use the term validity to refer to the approximate truth of an inference. When we say something
is valid, we make a judgment about the extent to which relevant evidence supports that inference
as being true or correct.”

Human judgment always factors into validity assessments, making absolute certainty impossible. Validity
judgments should be seen as tentative due to varying degrees of validity. It's crucial to note that validity is
about inferences, not methods or designs. Different circumstances can affect the validity of inferences from
the same design or method. For instance, even a randomized experiment doesn't guarantee a valid inference
about a causal relationship due to various factors like attrition, low power, improper statistics, or sampling
error.

Methods don't exclusively correspond to one validity type; using a method might affect multiple types of
validity simultaneously. For example, while a randomized experiment can improve internal validity, it might
hinder external validity. Practical design choices often have unforeseen consequences for validity, creating
new problems despite solving existing ones.

Validity and Truth: According to the authors, the concept of validity intertwines with truth in various
philosophical theories: correspondence, coherence, and pragmatism. Correspondence involves matching
claims with the world, coherence involves consistency within a set of claims, and pragmatism focuses on
usefulness. However, philosophers haven't reached consensus on a single theory of truth. In scientific practice,
validity relies on empirical evidence, coherence with existing theory, and pragmatic utility, blending elements
from these philosophical theories. Social and psychological factors significantly influence what is accepted as
truth in science. While the book doesn't extensively cover social theories of truth, it acknowledges truth as a
social construct shaped by external influences. The theory of validity in this book aims to be evaluative in
guiding what should be accepted as true, while acknowledging the role of external factors shaping what is
accepted as true in practice.

Validity Types: The concept of validity has evolved through the work of Campbell, Stanley, Cook, and
Campbell, leading to the typology of four key validity types: statistical conclusion, internal, construct, and
external validity. Initially, Campbell focused on internal and external validity, relating to cause-effect
relationships and generalizability across populations, settings, and variables. Cook and Campbell expanded
this typology, introducing statistical conclusion validity and emphasizing constructs and generalizations in
their definitions. Statistical conclusion validity pertains to inferences about the correlation between treatment
and outcome, while internal validity addresses whether observed covariation implies a causal relationship

1Samhita, L., & Gross, H. J. (2013). The “Clever Hans Phenomenon” revisited. In Communicative & Integrative Biology
(Vol. 6, Issue 6, p. e27122). Informa UK Limited. https://doi.org/10.4161/cib.27122
9
between variables. Construct validity evaluates the degree to which inferences correspond to higher-order
constructs, extending its scope to cover persons and settings beyond treatments and observations. External
validity, initially focusing on generalizing causal relationships across populations and settings, now includes
treatments and observations in its definition to assess causal relationships across variations in persons,
settings, treatment variables, and measurement variables. The rationale for this typology lies in addressing
fundamental questions researchers face when interpreting causal studies. These questions revolve around the
extent of covariation between presumed causes and effects, the causal relationship between variables, the
involved constructs, and the generalizability of embedded causal relationships across various factors. While
interconnected, each question requires distinct reasoning and analysis. However, it's essential to remember
that while a validity typology aids in design, critical analysis and logical reasoning are indispensable in any
specific case.

Threats To Validity: Threats to validity are factors that can introduce errors in inferences about covariance,
causation, constructs, or the generalizability of causal relationships. These threats are identified through a
combination of conceptual and empirical processes and can vary in relevance across different contexts.

Þ Design controls are recommended as the primary method to mitigate threats by minimizing their number and
plausibility. The book umderlines the importance of conducting studies with robust design controls, especially
through elements of experimental design that enhance causal inferences. However, some threats cannot be
entirely eliminated through design controls, and in such cases, the approach involves identifying and exploring
the role of the threat in the study by addressing critical questions about its application, plausibility, and potential
impact on observed effects.
Þ Post hoc criticisms: While post hoc criticisms often reveal threats, the authors advocates for anticipating threats
before a study begins. If a threat is anticipated but cannot be controlled through design, direct measurement is
recommended to assess its actual operation in the study. However, caution is advised in using direct measures
of threats in statistical analyses due to technical reasons, with a preference for design over statistical control.
Ruling out threats to validity is seen as a falsification enterprise, recognizing the fallible nature of knowledge,
judgment, and measurement, as outlined in Chapter 1's challenges.

Statistical Conclusion Validity (SCV): is about making sure a research accurately captures the relationship
between two things. SCV focuses on two crucial statistical inferences in causal studies: whether a presumed
cause and effect actually vary together and how strong that relationship is. Errors can happen—we might
think there's a connection when there isn't (Type I error) or miss a relationship that exists (Type II error). Also,
we could get the magnitude of this relationship wrong, either overestimating or underestimating it, affecting
how confident we are in our findings. While qualitative analyses matter, this chapter sticks to classic statistical
methods for measuring and understanding these connections. It starts with explaining how statistical
measures work and then dives into the specific things that could go wrong with these inferences.

Null Hypothesis Significance Testing (NHST): Critics argue that Null Hypothesis Significance Testing
(NHST), the conventional method for assessing cause-and-effect relationships, faces significant issues,
including misinterpretation and a limited ability to accurately reveal effect sizes. There's a growing push
among scientists to move away from NHST and adopt reporting methods centered on effect sizes with
confidence intervals. This approach aims to provide a clearer understanding of the magnitude of relationships
and the precision of effect size estimates. While there's a consensus against completely eliminating NHST,
the prevailing view is to reduce its role. Effect sizes and confidence intervals offer more comprehensive

10
information, emphasizing precision and accounting for uncertainty, in contrast to NHST's binary conclusions.
The evolving discourse suggests a diminishing role for traditional NHST in social science research due to its
limitations, advocating for nuanced result interpretations that acknowledge statistical uncertainty and
methodological limitations.

Threats to Statistical Conclusion Validity: In examining Statistical Conclusion Validity, a crucial aspect
involves recognizing and mitigating threats that can distort inferences about variable relationships, potentially
compromising research accuracy. These threats include low statistical power, arising from inadequate sample
sizes, which may lead to erroneous conclusions about the absence of significant relationships between
variables. Violated assumptions of statistical tests contribute to inaccurate estimations of effect sizes and
significance levels. Fishing and the Error Rate Problem involve repeated tests for relationships without proper
correction, potentially inflating statistical significance. Unreliable measures introduce measurement errors,
weakening the observed relationships between variables. The restriction of range in a variable can diminish its
relationship with another variable. Inconsistencies in treatment implementation may lead to an
underestimation of treatment effects. Extraneous variance in the experimental setting introduces errors that
make effect detection more challenging. Heterogeneity of units within groups can inflate error variance,
impacting the detection of relationships. Finally, inaccurate effect size estimation occurs when certain statistics
systematically overestimate or underestimate the size of an effect. Addressing these threats is crucial for
maintaining the validity of research conclusions.

The Problem of Accepting the Null Hypothesis: Recognizing when to declare "no effect" is crucial,
especially in scenarios where anticipated changes align with no impact or multiple experiments yield
inconclusive results. Proving an exact zero effect is challenging due to potential small effects with robust
methodologies. Maximizing statistical power is essential, and researchers can evaluate the feasibility of
conducting more powerful experiments using outlined strategies. Identifying a worth-pursuing effect size sets
a benchmark for innovation evaluation, even with a partial effect. Presenting absolute treatment effect
magnitudes helps readers assess practical significance or the need for further exploration after a non-
significant effect. Equivalency testing provides an alternative to traditional hypothesis testing, determining if
observed effects fall within an acceptable range. Lastly, quasi-experimental analyses explore larger effects
under specific conditions, with caution due to potential biases and chance capitalization. If these analyses fail
to reveal interesting relationships, confidence in pursuing the effect diminishes.

Internal Validity: Internal validity refers to assessing whether the observed relationship between variables A
and B indicates a causal link from A to B in the form they were manipulated or measured. This inference
requires demonstrating that A preceded B in time, showing their covariation (a facet covered under statistical
conclusion validity), and ruling out alternative explanations for this relationship. Experimental designs usually
solve the issue of causal order by manipulating A before measuring B, but nonexperimental research,
especially cross-sectional studies, faces challenges in establishing temporal precedence

The term "internal validity" has been used in various ways in social sciences, deviating from its original concept
described by Campbell. To address these misconceptions, Campbell proposed relabeling it as "local molar
causal validity," emphasizing causal inferences limited to specific contexts, treatments, settings, and
individuals studied.This term underscores that experiments test complex treatment packages, acknowledging
their multivariate nature. For instance, psychotherapy involves various verbal and nonverbal interventions,

11
professional cues, physical environments, etc. When someone undergoes psychotherapy in an experiment,
they are exposed to this entire package, not just the parts the researcher intends to test. Thus, the causal
inference from an experiment assesses the impact of the entire treatment package. Despite breaking down
treatments into individual parts, even these molecular aspects comprise numerous components. Framed as
"local molar causal validity," internal validity evaluates whether a multivariate treatment caused a difference
in a specific variable within the sampled units, settings, and time frames of a study.

Threats to internal validity: Threats to internal validity are those other possible causes-reasons to think that
the relationship between A and B is not causal, that it could have occurred even in the absence of the
treatment, and that it could have led to the same outcomes that were observed for the treatment. Threats to
internal validity pose challenges to establishing a strong causal relationship between variables. Researchers
must recognize and address these threats to ensure the reliability of study results.

One major threat is selection bias, where groups differing before treatment impact outcome attribution.
Random assignment mitigates this bias, but its effectiveness varies, particularly in quasi-experiments.
Historical events between treatment initiation and posttest can confound outcomes, requiring meticulous
group selection and consistent testing schedules. Natural participant changes over time, termed maturation,
pose threats, which researchers address by ensuring group similarity and minimizing individual variations.
Regression artifacts, especially in quasi-experiments, may be mistaken for treatment effects, demanding
cautious result interpretation and diagnostic tests for regression.Attrition, or experimental mortality,
introduces biases when participants fail to complete measures. Addressing post-treatment selection bias
becomes crucial, especially considering that attrition is not controlled by random assignment.Taking tests can
influence subsequent scores, introducing practice or reactivity effects. Awareness of testing effects is vital,
employing techniques like item response theory to minimize them. Changes in measuring instruments over
time may mimic treatment effects, emphasizing the need for instrument consistency throughout a study.

Þ In focusing on internal validity to ensure observed outcomes result from testing, random assignment in
experiments plays a crucial role. It helps eliminate biases, reducing the chance of other issues affecting results.
Challenges arise in randomized experiments, such as differential attrition or testing differences between groups.
Researchers monitor and correct these issues, employing advanced techniques for resolution.
Þ Quasi-experiments, with systematic differences between groups, pose challenges to internal validity.
Researchers modify study designs, reduce threats, and minimize attrition, cognizant of potential alterations to
the research question. Explicitly identifying and evaluating threats is essential, whether quantitative or
qualitative, with reliance on randomized experimental design for ensuring internal validity, considering context-
specific threats.

The relationship between internal validity and statistical conclusion validity is intricate. While statistical
conclusion validity focuses on errors in assessing statistical covariation, internal validity is concerned with
errors in causal reasoning. Even flawless statistical analyses do not guarantee correct causal conclusions. The
reliance on randomized experimental design, especially with random assignment, enhances the reliability of
findings by minimizing biases and threats to internal validity. In the context of quantitative and qualitative
experiments, the connection between internal validity and statistical conclusion validity persists. In
quantitative experiments, where interventions and outcomes are measured quantitatively, the two are closely
linked. However, qualitative experiments, such as Sherif's Robber's Cave Experiment, demonstrate that even
without quantitative data, internal validity can be maintained through qualitative assessments of covariation.

12
Campbell and Scriven argue that both quantitative and qualitative research share similar logical requirements
for causal inferences. While recognizing the limited circumstances for valid causal inferences from case
studies, they reject the notion that causation necessitates quantitative treatments or outcomes.

Conclusion

In conclusion, our exploration of validity emphasizes drawing accurate inferences from empirical evidence.
Validity revolves around inference accuracy, influenced by human judgment, making absolute certainty
elusive. Validity judgments are provisional due to varying inherent accuracy.

Exploring statistical conclusion validity (SCV) highlighted the need for accurate depiction of variable
relationships. Traditional methods like null hypothesis significance testing (NHST) have limitations, leading to
a shift towards using effect sizes and confidence intervals for a better understanding with acknowledgment
of uncertainty. Internal validity, vital for establishing causal links, faces several threats that researchers must
tackle to ensure study reliability. The close relationship between internal validity and statistical conclusion
validity remains crucial, especially in randomized experimental designs, to minimize biases. This connection
is significant across various research methods, stressing the importance of logical requirements for valid
causal inferences.

Throughout this exploration, it has become evident that validity isn't a static concept but a nuanced and
evolving framework that incorporates empirical evidence, coherence with existing theory, and pragmatic
utility. Validity, therefore, serves as a guiding principle in determining what should be accepted as true,
acknowledging the influence of social and psychological factors in shaping scientific truths. This nuanced
understanding of validity and its associated threats is pivotal for researchers seeking to draw accurate
inferences from empirical evidence.

13
Harvard Business
Review
Strategic Insights into Business Experimentation: A Review of HBR's Guides on
Designing and Analyzing Smart Experiments, Including a Refresher on
Randomized Controlled Experiments.

14
Introduction

In many organizations, the impulse to implement new ideas often precedes any real evidence to support
them. Initiatives are undertaken, offerings are tweaked, distribution methods are experimented with, and
work processes are altered, all based primarily on gut feelings or what seems like common sense. Thomas
H. Davenport highlights this prevalent issue, stating, "Too many business innovations are launched on a
wing and a prayer – despite the fact that it’s now reasonable to expect truly valid test." This approach,
unfortunately, leads to decisions veiled in the language of science but lacking investigative rigor. These
pseudo-experiments fall short of being truly investigative and, consequently, are likely to yield incorrect
assumptions, with minimal learning derived from the process itself. (Davenport, HBR, February 2009, p.
70).

In the pursuit of informed decision-making, organizations are increasingly recognizing the need for a
shift from intuition-based approaches to evidence-based methodologies. As Davenport notes, “the
triumphs of testing occur in strategy execution, not strategy formulation." He emphasizes that
scientifically valid experiments are well-suited for assessing the potential impact and value of tactical
changes, such as new store formats, marketing promotions, or service processes.

Too many business innovations are launched on a wing and a prayer – despite the fact that it’s now
reasonable to expect truly valid test.

Building upon this, Hauser and Luca contribute by highlighting the transformative potential of field
experiments in organizational decision-making. They stress the importance of crafting precise questions
aligned with overarching goals and the significance of large-scale interventions. Their perspective
complements Davenport's by underlining the need for organizations to invest in randomized evaluations,
providing a game-changing advantage in various business domains.

Amy Gallo's insights add another layer to the discussion, emphasizing the necessity for data in making
smart decisions at work. In her words, "There is a spectrum of experiments that managers can do from
quick, informal ones to pilot studies, to field experiments, and to lab research." She particularly highlights
the significance of randomized controlled experiments as one of the more structured and powerful
approaches.

Furthermore, Gallo stresses the spectrum of experiments, ranging from quick, informal ones to more
structured experiments like randomized controlled experiments. Her insights reinforce the idea that the
choice of experimentation method depends on factors such as specific goals, desired accuracy, and
available budget.

15
Hauser and Luca also emphasize the importance of commitment to a plan and meticulous execution.
They state, "Commit to a plan, and stick to it. Before you run an experiment, lay out your plans in detail."
This includes considerations such as the number of observations, duration of the experiment, and
variables to be collected and analyzed.

Conclusion

Collectively, these articles advocate for a paradigm shift toward evidence-based decision-making through
rigorous experimentation. Davenport's call for a "test and learn" culture, Hauser and Luca's emphasis on
large-scale interventions and randomized evaluations, and Gallo's focus on the structured approach of
randomized controlled experiments converge to provide organizations with a roadmap for building a
culture of testing and learning. The central theme that emerges is the iterative nature of experimentation,
driving continuous learning within organizations. By embracing scientifically valid experiments and
avoiding common pitfalls, businesses can challenge assumptions, improve practices, and ultimately make
more informed, evidence-based decisions.

Bibliography
1. Davenport, T. H. (2009). How to design smart business experiments. In Strategic Direction (Vol. 25, Issue 8). Emerald.
https://doi.org/10.1108/sd.2009.05625had.004
2. Hauser, O., & Luca, M. (October 29, 2015). "How to Design (and Analyze) a Business Experiment." Harvard Business
Review. Available at https://hbr.org/2015/10/how-to-design-and-analyze-a-business-experiment
3. Gallo, A. (March 30, 2016). "A Refresher on Randomized Controlled Experiments." Harvard
Business Review. Available at https://hbr.org/2016/03/a-refresher-on-randomized-controlled-experiments

16
Exemplary
Research
Exploring diverse consumer behaviors, these studies reveal intrinsic
motivations driving choices. From brand exposure's automatic effects on
behavior to vicarious losing affecting eating habits, and choosing green
products for status, these studies collectively emphasize social and
psychological influences on consumer decision-making.

17
Consumer goal pursuit
By Juliano Laran,2016.

In this article, Juliano Laran explores how consumers go after their goals, diving into two main approaches:
conscious and unconscious goal pursuit. When it comes to conscious pursuit, people are aware of their
goals, track their progress, and plan their next moves. Unconscious pursuit happens when cues from the
environment activate goals without consumers realizing it, guiding their choices toward objectives they
might not be consciously thinking about.

Laran reviews recent research findings in both conscious and unconscious goal pursuit. For conscious
pursuit, factors like assessing progress and considering attainability are crucial motivators. How close
individuals feel to achieving a goal significantly impacts their motivation. The article also discusses various
factors influencing consumer motivation, such as goal nature, visualization, attention, timing,
interruptions, and consequences of actions.

On the flip side, unconscious goal pursuit involves consumers being influenced by environmental cues
without consciously registering the triggered goal. Studies reveal that exposure to stimuli linked to
specific goals affects consumer behavior. However, consumers aren't passive; factors like context novelty,
perceived intent, and alignment with personality can influence how effective unconscious goal pursuit is.

In conclusion, the article proposes an integrative perspective on conscious and unconscious goal pursuit.
Instead of treating these processes in isolation, Laran suggests understanding how they interact to
influence consumer behavior. Both conscious and unconscious processes bring unique strengths, and
future research should delve into when these systems complement or hinder each other in achieving
consumer goals. Laran's review provides a comprehensive look into how consumers pursue goals,
unraveling the intricate interplay between conscious and unconscious processes driving consumer
behavior.

Automatic Effects of Brand Exposure on Motivated Behavior: How Apple

Makes You “Think Different”
By Grainne M. Fitzsimons, Tanya L. Chartrand, Gavan J. Fitzsimons,2017.

This study explores the automatic behavioral effects of brand exposure, drawing parallels with social cues'
impact on our actions. It specifically explores whether brands, like social cues, can trigger behavior
changes, known as social priming effects.

The research reveals that exposure to certain brand logos, such as Apple, can prompt specific behaviors.
Participants primed with Apple logos tend to exhibit more creativity compared to those exposed to IBM
logos or control groups. Similarly, Disney logos lead to more honest behavior compared to E! logos.

The experiments investigate whether brand exposure can lead to goal-directed behavior, emphasizing
that brands play a role in shaping general behavior, going beyond traditional consumer decisions. The
study proposes a framework for predicting when brand exposure triggers goal-directed behavior,

18
highlighting the importance of perceiving a gap between current and desired states associated with
positive emotions.

The experiments explore the impact of brand priming on behavior using different brands and behavioral
measures. For instance, Experiment 1 shows that participants primed with Apple generated more creative
uses compared to those primed with IBM. Experiment 2 reveals that participants primed with Disney
Channel behaved more honestly than those primed with E! Channel. Experiment 3 emphasizes the
specificity of brand priming effects, demonstrating that brands influence behavior primarily in individuals
possessing a relevant goal, reinforcing the role of motivated processes.

The overall conclusion is that brand exposure goes beyond influencing store choices; it can shape
creativity, honesty, and trigger goal-directed actions. The study emphasizes the need to understand both
cognitive and goal-driven pathways in consumer actions. The role of motivation emerged notably in the
experiments, highlighting the intricate relationship between brands, motivation, and subsequent actions,
providing valuable insights for marketing strategies and consumer research.

From Fan to Fat? Vicarious Losing Increases Unhealthy Eating, but Self-
Affirmation Is an Effective Remedy
By Yann Cornil and Pierre Chandon,2013.

Yann Cornil and Pierre Chandon conducted a comprehensive study exploring the intricate link between
sports outcomes and fans' dietary choices. Focusing on the National Football League (NFL) seasons in
2004 and 2005, the researchers aimed to decipher how victories and defeats of local football teams
affected the eating behaviors of their fans.

The study, encompassing 475 NFL games and 30 teams, selected cities based on their team performances,
assuming residents generally support their local team. Data on food consumption were collected from a
diverse sample of Americans residing in major U.S. metropolitan areas. The researchers utilized two
independent rankings to gauge the level of attachment to an NFL team in each city.

The findings revealed a compelling connection between game outcomes and dietary habits. After a
defeat, saturated-fat consumption increased significantly by 16%, while after a victory, it decreased by
9%. Similar patterns were observed for total food-calorie consumption, showing a 10% increase after a
defeat and a 5% decrease after a victory. Notably, these effects were more pronounced in cities with the
most devoted fans, especially after narrow defeats or large victories, emphasizing the emotional impact
of game results.

Furthermore, the study extended its exploration to the psychological aspect of fan behavior. The research
underscored the emotional responses to game results and their potential influence on self-regulation
abilities related to food consumption. The study also highlighted the potential mitigating effect of self-
affirmation, emphasizing the psychological aspects of fan behavior in response to sports outcomes.

A subsequent study by Cornil and Chandon involved French adults interested in sports, tasked with
writing about a victory or defeat of their favorite team or athlete. After this, participants engaged in an

19
unrelated activity involving access to snacks. Results indicated that after a defeat, participants consumed
more calories from saturated fat and added sugar compared to after a victory. Notably, spontaneously
affirming one's core values seemed to counteract the effects of a defeat on unhealthy consumption,
pointing to the role of self-affirmation.

In Study 3, participants watching highlights of soccer games showed that watching a defeat led to
unhealthier consumption intentions compared to watching a victory or a neutral game. Self-affirmation
completely eliminated the effects of a defeat on consumption intentions. These findings further solidified
the notion that people tend to eat less healthily after watching their favorite team lose, highlighting the
potential role of emotions, identification, and self-threats in influencing post-game eating habits.

In summary, these studies collectively illuminate the profound impact of sports outcomes on dietary
choices. The emotional and psychological dimensions of fan behavior, as well as the mitigating effect of
self-affirmation, reveal valuable insights into the complex relationship between sports experiences and
eating habits.

Going Green to Be Seen: Status, Reputation, and Conspicuous Conservation

By Vladas Griskevicius, Joshua M. Tybur, Bram Van den Bergh,2010.

This study, challenges common explanations for choosing eco-friendly products, focusing on the Toyota
Prius and applying costly signaling theory. The research suggests that, despite higher costs, people opt
for green products as a form of altruism, signaling a willingness to incur costs for the benefit of society
and the environment.

The Toyota Prius is highlighted as a prime example, where owners see their choice as making a statement
about their values, emphasizing the connection between green choices and personal values. The study
introduces the concept of "competitive altruism," proposing that status motives play a significant role in
driving the preference for green products, even when more luxurious non-green alternatives exist.

A key finding is that conservation, rather than being solely driven by environmental concerns or economic
factors, is socially motivated. The study emphasizes the importance of tapping into the social and
reputational aspects of conservation. Appeals that highlight the behaviors of others are found to be more
effective in promoting environmentally conscious behavior.

The relationship between conservation, reputation, and status is explored, suggesting that going green
can signal prosocial behavior, contributing to a positive reputation. Individuals engaging in prosocial
behaviors, including environmental conservation, are viewed as more trustworthy and desirable. Self-
sacrifice for the benefit of a group enhances one's status within that group.

Three experiments delve into the interplay between status motives and green choices. Experiment 1
reveals that activating status motives increases the preference for less luxurious green products,
supporting the costly signaling theory. Experiment 2 extends this by showing that the influence of status
motives is contingent on the cost associated with green choices. Experiment 3 adds nuance, indicating
that individuals with high goal alignment related to environmental conservation are more susceptible to
the influence of status motives.

20
The study concludes by emphasizing the multifaceted nature of pro-environmental choices. It suggests
that understanding the social motives behind green choices offers a more effective strategy for
promoting eco-friendly behavior. Marketers and policymakers are encouraged to leverage insights into
individuals' desires for positive reputations and status when designing interventions to encourage
sustainable consumer choices.

The research has implications for marketing and policy. Marketers are advised to link green products to
status, especially when the products are relatively expensive. Strategies emphasizing the social prestige
associated with choosing environmentally friendly options, particularly in public settings, are
recommended. Pricing strategies for green products should consider both the price and individuals'
status motives. The study also emphasizes tailoring interventions based on the visibility of choices,
recognizing the distinction between public and private settings.

Despite valuable insights, the study acknowledges limitations, primarily focusing on specific product
choices in controlled settings. It calls for future research to explore real-world environmental decisions,
diverse populations, and the long-term impact of interventions. Overall, the research contributes to a
deeper understanding of the psychological mechanisms that drive sustainable behavior and offers
practical implications for promoting environmental conservation in a socially prestigious manner.

Conclusion
Consumer behavior, as explored in these studies, is deeply influenced by various psychological factors.
Goals drive our choices, whether apparent or hidden, while brand exposure subtly shapes our preferences
and self-perception. Additionally, observing others' experiences can impact our behavior, as seen in how
vicarious losses influence eating habits. Moreover, the pursuit of status and the desire for a positive
reputation can drive environmentally friendly choices, revealing the social motives behind such decisions.
Understanding these interconnected psychological influences is key to crafting effective strategies that
resonate with consumers' goals and aspirations.

Each study offers a unique angle on consumer behavior—be it goal pursuit, brand influence, vicarious
experiences, or the social signaling of environmental choices. They collectively shed light on how
psychological triggers, aspirations, and social dynamics play significant roles in shaping what consumers
buy and why. These studies paint a complex yet intriguing picture of the myriad factors influencing our
choices when we shop or make decisions related to brands and products.

Bibliography
1. Cornil, Y., & Chandon, P. (2013). From Fan to Fat? Vicarious Losing Increases Unhealthy Eating, but Self-Affirmation Is an
Effective Remedy. In Psychological Science (Vol. 24, Issue 10, pp. 1936–1946). SAGE Publications.
https://doi.org/10.1177/0956797613481232
2. Fitzsimons, G. M., Chartrand, T. L., & Fitzsimons, G. J. (2008). Automatic Effects of Brand Exposure on Motivated Behavior:
How Apple Makes You “Think Different.” In Journal of Consumer Research (Vol. 35, Issue 1, pp. 21–35). Oxford University
Press (OUP). https://doi.org/10.1086/527269
3. Griskevicius, V., Tybur, J. M., & Van den Bergh, B. (2010). Going green to be seen Status, reputation, and conspicuous
conservation. In Journal of Personality and Social Psychology (Vol. 98, Issue 3, pp. 392–404). American Psychological
Association (APA). https://doi.org/10.1037/a0017346
4. Laran, J. (2016). Consumer goal pursuit. In Current Opinion in Psychology (Vol. 10, pp. 22–26). Elsevier BV.
https://doi.org/10.1016/j.copsyc.2015.10.015

21