Factorial Validity

Dr. Gilles Gignac

1
Aims
 Cronbach’s alpha assumption revisited
 Introduce factor analysis
 Unidimensional
 Multidimensional (uncorrelated)
 Multidimensional (correlated)
 Conducting and Interpreting a PCA
 Scree plot
 Communalities
 Eigenvalues
 Component Loadings
 Simple Structure
 Component correlations
2
Previously…
 Internal consistency reliability is typically estimated with
Cronbach’s alpha.
 Cronbach’s alpha assumes unidimensionality
 Composite scores represent one dimension or one construct.
 How does one determine whether a group of items
actually measure a single dimension?
 Relatedly, how do we justify adding two or more items
together?
 We use a technique known broadly as ‘factor analysis’.
 Also known as ‘data reduction analysis’.

3
Example
 Consider the items below which form a fictitious, 6-item test.
 1: I enjoy socialising with a large number of people
 2: I spend time pondering philosophical questions
 3: I’m happy to lead a discussion.
 4: I like to entertain novel solutions to intellectual problems.
 5: I have a large social group.
 6: Spending a Friday night with a good book is fine by me.

 Can you detect a pattern?

 Items 1, 3, and 5 appear to measure a socialisation dimension
 Items 2, 4, and 6 appear to measure an intellect dimension.
 Do you think socialisation and intellect would be correlated dimensions?

4
Student Opinion Scale

Importance Items:
1, 3, 4, 5, 8

Effort Items:
2, 6, 7, 9, 10

Two correlated dimensions?

Factorial Validity (and Factor Analysis)
 Factorial Validity is relevant to determining whether the scores of a
measure correspond to the number and nature of dimensions of
theorized dimensions that underlie the construct of interest.

 Researchers use a data analytic technique known as factor analysis to

help determine the number and nature of dimensions associated with
the scores derived from a test.

6
Factorial Validity
 Factorial validity pertains to the internal structure of test scores.
 A test’s internal structure is the way that the parts of the test are
related to each other.
 The actual (empirical) structure of the test should match the
theoretical or intended structure of the test.
 In a very simplistic sense, factorial validity helps specify what the
test measures.
 That is, with respect to the number of dimensions, and the definition
of those dimensions.
 The definitions of the dimensions are determined by which items
“load” onto which dimension.

7
Factorial Validity
 To determine the number of dimensions that underlie an inventory, as
well as their definitions, we (can) use a data analytic technique
known as factor analysis.

 Factor analysis…
 1: Helps us clarify the number of factors within a set of items (or indicators)
 2: Helps us determine the nature of the associations among the factors
 3: Helps us determine which items are linked to which factor, which facilitates
the interpretation of those factors

8
Measurement
 Typically, when we attempt to measure an attribute, we try
to do so in such a way that the score represents a single
attribute.
 Consider the example above based on the six-items. If I
calculated a total score based on those six items, what
would that composite score mean?
 It would be a compromised score, because the items do
not all measure the same dimension.
 In practice, we try to avoid such situations.
 We want composite scores that are relatively “factor
pure”.

9
Dimensionality
 As a general statement, there are three different types of tests:
 1: Unidimensional test
 Consists of items that all measure one, single factor
 2: Multidimensional test (uncorrelated)
 Consists of items which measure two or more dimensions which are
unrelated to each other
 3: Multidimensional test (correlated)
 Consists of items which measure two or more dimensions which are
correlated with each other (positively or negatively)

10
Unidimensional Tests
 They include items that reflect only a single attribute
 The items are not affected by other attributes
 Consider a multiple choice final exam
 Students only get one score on the final exam.
 This presumes that the test is unidimensional.
 It may be reasonable to consider that the items all measure
a single dimension – “knowledge of the subject matter
covered in class”.
 However, it is not necessarily the case.

11
Unidimensional Tests
 Consider the Effort dimension from the Student Opinion
Scale
 There are 5 items relevant to the amount of effort applied
when completing a test.
 2. I engaged in a good effort throughout this test.
 6. I gave my best effort on this test.
 7. While working on this test, I could have worked harder.*
 9. I did not give this test my full attention while completing it.*
 10. While taking this test, I was able to persist to completion of
the task.
 Theoretically, we’d expect those 5 items to measure a
single dimension: test-taking effort.
12
Unidimensional Model: Effort

 This is a model of a unidimensional test

 Each item is linked to one attribute only
 The degree to which each indicator is linked to the attribute
can be estimated with factor analysis or component analysis.

13
SOS: Effort Component Solution

 On the basis of the extraction of a single component (i.e. one factor), all
five items were found to be loading onto the component.
 Item 6 loaded the most strongly at .840
 Item 10 loaded the most weakly at .426
 A loading is like a standardized beta-weight (from regression)
14
Effort Component Solution
 All 5 items contributed variance to the Effort component.

15
Inter-Variable Correlations: Effort Items

 Component and factor analyses rely

on correlations between variables.
 Assessing inter-item correlations is
crucial prior to analysis.

 Correlation ranges from 0.078 to 0.626.

 Higher scores on one item link to higher scores on others, indicating consistency
in rated test-taking effort.
 One correlation non-significant (p = .248).
 Majority of correlations significant (p < .001).

 A large proportion of significant correlations suggests possibly justified data

reduction analysis.
16
Bartlett’s Test of Sphericity
 Sophisticated method to assess correlation matrix
suitability for data reduction.
 Tests if the matrix is an identity matrix (zero correlations).
 A significant test result suggests non-zero correlations.
 Indicates potential for underlying dimensions suitable for
factor analysis.
 In this analysis, Bartlett’s test: χ2(10) = 324.00, p < .001.
 Result implies that data reduction analysis may be warranted.
 With samples of 500+ participants, significance is likely
regardless of actual correlations.

17
KMO’s Measure of Sampling Adequacy
 KMO values range from 0 to 1
 Higher values indicate greater suitability for factor analysis.
 KMO Guidelines (Kaiser & Rice, 1974)
 < 0.60: Unacceptable for data reduction.
 = 0.65: Minimally acceptable, interpret with caution.
 ≥ 0.70: Somewhat respectable, more reliable for factor analysis.
 ≥ 0.80: Respectable, suitable for meaningful analysis.
 ≥ 0.90: Marvelous, excellent data quality for factor analysis.
 KMO for test-taking effort items: .794.

18
Number of Dimensions: Scree Plot
 Choosing the number of dimensions for data reduction: Critical decision
 Various methods exist, each with unique merits.
 Scree plot: A graphical tool for visualizing eigenvalues in descending
order.
 Maximum eigenvalues equal to the number of observed variables
(example: 5 items = 5 eigenvalues).
 Eigenvalues indicate the variance accounted for by each component.
 Eigenvalues start high and taper off, resembling a rugged hillside
("scree").
 The "elbow" point indicates a natural cutoff for component
significance.
 Larger eigenvalues account for more variance, indicating significant
components.

19
Scree Plot: Example
 Scree Plot from SOS: Effort Items (5 items)
 First eigenvalue significantly exceeds 2.0, distinguishing
it from others.
 Notable drop between the first and second eigenvalues.
 Existence of an additional, less dominant 2nd factor?
 Analysis can be subjective, leading to expert
disagreement.
 Need a more objective method.

20
Parallel Analysis
 Compares actual data eigenvalues to
randomly generated data set eigenvalues
 Eigenvalues from random data represent the
null hypothesis.

 Component eigenvalues greater than the

95th percentile of random data
eigenvalues are statistically significant.
 Provides an empirical criterion; less subjective
than scree plot.

 Only the first component's eigenvalue

exceeds that of the corresponding random
data.
 Suggests one meaningful dimension
(Effort?) to be extracted for data reduction.

21
Multidimensional Model: Effort & Importance
 Multidimensional models are more
prevalent in psychology
 SOS aimed to measure Importance
and Effort
 10 items total (five per dimension).
 Dimensions are inter-correlated

 Perfect zero loadings on unintended dimensions are unrealistic.

 Focus on whether cross-loadings are sufficiently small.

22
 Observed Variable Inter-Correlations
 Data reduction is
fundamentally based on
the correlations
between the observed
variables.
 Good practice to
examine them before
data reduction.
 Many correlations
statistically significant
(p < .05), indicating
potential dimensional
structure.
 Not all correlations significant, e.g., item 3 (SOS_3_r) vs. item 5 (SOS_5) with p = .204.
 All correlations are positive, hinting at one or possibly two dimensions.

23
Bartlett’s Sphericity Test & KMO Index
 Bartlett’s test: Statistically significant (p < .001).
 KMO Measure of Sampling Adequacy (MSA):
 Overall MSA = .77, exceeding .65 threshold.
 Both tests validate the suitability of data reduction for the
correlation matrix.

24
Parallel Analysis: SOS 10 Items
 Two eigenvalues significantly exceed those of randomly generated
data, indicating statistical significance.
 Will proceed with extracting two dimensions based on analysis
outcome.

25
Unrotated Component Solution
 All items show positive loadings on Component 1
 "Global test-taking effort" dimension?

 Mixed loadings on Component 2:

 Negative for items 2, 6, 7, 10 (effort); positive for
others (importance).
 Possible interpretation: A "low effort/high
importance" dimension?

 All items significantly load on both dimensions,

against the preference for dimension-specific
purity.
 Implement rotation to better align items with
specific dimensions.

26
Rotation in Data Reduction Analysis
 A technique applied after initial component or factor extraction.
 Aims to simplify and clarify data structure.
 Benefits of Rotation:
 Enhances variable distinction by aligning each more strongly with a single
dimension.
 Achieves a "simple structure" for easier interpretation and analysis.

 Goal of Simple Structure:

 Each variable should load significantly on one dimension & minimally on others.
 Facilitates clear identification and description of extracted components.

27
Understanding Rotation
 Types of Rotation
 Orthogonal: Keeps factors uncorrelated.
 Oblique: Allows factors to correlate.

 Purpose of Rotation
 To enhance interpretability by maximizing high loadings and minimizing low
loadings on each factor.

 Nature of Rotation
 Does not alter data; shifts perspective for clearer variable associations.
 A legitimate technique to simplify factor analysis, not data manipulation.

28
The Case for Oblique Rotation
 Orthogonal rotation is simpler but considered outdated
 Imposes artificial constraints on dimension correlations.

 Advantages of Oblique Rotation

 Oblique rotation offers flexibility
 More nuanced approach as it allows natural dimension correlations.

 May yield correlated or uncorrelated dimensions for optimal

structural simplicity.
 Combines the benefits of both rotation methods

29
Jamovi and Rotation Methods
 Orthogonal Methods in Jamovi
 Varimax and Quartimax available
 Not preferred due to limitations in capturing natural factor
correlations.
 Oblique Methods in Jamovi
 Promax, (Direct) Oblimin, and Simplimax
 Oblique solutions produce a ‘Pattern Matrix’
 Technical term to distinguish from ‘Component Matrix’ or ‘Factor Matrix’
 Emphasis on their suitability for contemporary data analysis needs.

30
Selecting Oblique Rotation Technique
 Popular Oblique Techniques
 Promax and Oblimin (Direct) Oblimin are the most used.
 Lack of unanimous preference due to inconsistent research findings.
 Recommendation to trial both (including Simplimax) to find the
most interpretable result.
 Iterative Application
 Valid to use multiple methods on the same dataset to discover the best solution.
 Not a compromise on analytical rigor but a strategy to improve result clarity and
interpretability.

31
 Rotation on SOS 10 Item Solution
 Conducted Oblimin and
Promax rotations on a 10-
item, two-component model.
 Results show minimal
differences (within |.02|) but
vary across datasets.
 Promax and Oblimin can yield
different solutions;
comparison is crucial for
clarity.
 Component 1: Dominated by 'effort' items with loadings between .53 and .82 (Promax).
 Component 2: Dominated by 'importance' items with loadings between .54 and .76.
 No loadings are exactly zero; focus on loadings' substantiality for meaningful variance
contribution.
32
Guidelines for Assessing Loadings
 Different criteria: individual items vs. composite scores.
 Different amounts of variance

 Individual items: Minimum loading of |.20|.

 Composite scores: Minimum loading of |.30|.
 Promax rotation shows item 10 with significant loadings on both
Effort (.53) and Importance (-.20) components (cross-loading).
 Negative loading on Importance hints at a nuanced relationship;
more persistence may imply less perceived importance?
 Calls for replication of results to confirm the validity of this
unexpected effect.

33
Correlation Between Components
 Oblique rotation reveals a .38 correlation between test-taking effort
(Component 1) and importance (Component 2) dimensions.
 Suggests a positive association; higher importance ratings relate to
increased effort.
 Cannot imply causality due to the study's observational nature.
 Stats programs do not test significance of factor loadings or
component correlations.
 Researchers rely on correlation strength and direction for
interpretive insights.

34
Introduction to Uniqueness values
 Indicates the variance of an item not explained by
the model.
 Lower values preferred, indicating significant
contribution to the model's dimensions.
 Item 6 has a uniqueness value of .30, suggesting a
moderate contribution.
 Item 10 shows the highest uniqueness at .75,
indicating the least contribution to the model.
 High uniqueness may signal poor fit, leading to
potential item removal considerations.
 Item 10’s low contribution highlighted by both high
uniqueness and lower factor loading.

35
Communality and Uniqueness in Data Reduction
 Communality (h2) represents the variance of an item contributed to
the model, opposite to uniqueness.
 Conversion between uniqueness and communality is straightforward
 h2 = 1 - uniqueness

 Item 6 with .30 uniqueness has .70 communality (1 - .30)

 For SOS h2 range: .25 (lowest, item 10) to .70 (highest, item 6).
 Guidelines for Evaluating Commonalities
 Minimum of .04 for individual items, and
 Minimum of .09 for composite scores.

 Rotation does not alter communalities, affirming its role in improving

interpretability without changing data structure.

36
Sums of Squared Loadings & Eigenvalues
 Effort (Comp. 1) has a larger eigenvalue
(2.77) than Importance (Comp. 2) at 2.16.
 Effort explains 27.67% of total variance
 Importance 21.64% of total variance.

 Jamovi: 'SS Loadings’

 Represents eigenvalues as the sum of
squared component loadings.
 Effort's and Importance's SS Loadings
align with their eigenvalues
 Interpretation of 'SS Loadings'
 Reflects the total variance each
component explains (rotated solution).
 2.73 close to 2.77; 2.14 close to 2.16
37
Application of Sum of Squared Loadings
 Calculate individual item uniqueness and communality.
 Example Calculation for SOS Item 2 (Loadings: .7631 and -.3260)
 Squaring loadings (.58 and .11) and summing yields a communality of .69.
 Subtracting from 1 gives a uniqueness value of .31, matching Jamovi.
 Approach consistently produces uniqueness values close to those reported by
Jamovi, considering rounding.

 Unrotated sum of
squared loadings for
Components 1 and 2
(3.48 and 1.45) used for
scree plot and parallel
analysis.

38
Putting it all Together
 A principal component analysis was conducted on the 10-item Student
Opinion Scale. Parallel analysis revealed two statistically significant
dimensions with 95% confidence. Following the extraction of two
components and an Oblimin rotation, five items pertaining to test-
taking Effort had significant loadings (.20 or greater) on Component 1
(rotated eigenvalue = 2.73, accounting for 27.67% of the variance), and
five items related to test-taking Importance had significant loadings on
Component 2 (rotated eigenvalue = 2.14, accounting for 21.64% of the
variance). Loadings for the Effort component ranged from .54 to .82,
and from .55 to .76 for the Importance component. Together, these
dimensions explained nearly half of the variance in item responses
(49.31%). The components showed a correlation of r = .34, suggesting a
positive association between levels of test-taking effort and
importance.

39
Introduction to Higher-Order Dimensions
 Higher-order dimensions can capture shared variance among scale dimensions.
 Example: the shared variance between Effort and Importance dimensions in the
Student Opinion Scale can form a 'Global Test-Taking Motivation' construct.

 Higher-order dimensions facilitate a

holistic understanding of the measured
construct
 May justify calculation of an
overarching motivation score.
 If loadings to Global dimension loadings
are appreciable.

40
Uncorrelated Dimension Models
 Certain constructs, like Extraversion and
IQ, are measured by sets of items
designed to assess specific attributes
independently.
 Specifies dimensions as uncorrelated,
ensuring items are uniquely associated
with a single attribute without overlap.
 Scores for each dimension (Extraversion,
IQ) are calculated separately to maintain
construct integrity.
 Combining scores into a single total score
is not appropriate due to the theoretical
and empirical distinction of the
constructs.

41
Component Vs. Factor Models
 Component Analysis
 Defines components based on observed
variables without assuming latent causes.
 Useful for simple data reduction without any
theoretical preconceptions.

 Factor Analysis
 Assumes latent variables cause observed items to correlate.
 Incorporates error terms, acknowledging variance not explained by the latent factor.

 Theoretical and Empirical Distinctions

 Theoretically, factor models infer latent constructs driving observed correlations.
 Component models do not presume a cause of the correlations between items.

 Factor models are preferred for identifying latent constructs.

 Component models are chosen for their simplicity in data reduction.
42
 Factor Versus Component Solutions
 Conducted Factor Analysis (MLE) and Component analysis on 10 SOS items.
 Same five items defined Effort (Factor 1) and Importance (Factor 2)
 Though similar, factor
loadings generally smaller
than component loadings
 Item 6's factor loading (.78)
vs. component loading (.82).
 No significant cross-
loading for item 10 on
Importance factor (loading
= -.07).

 Factor analysis: preferred method.

 Factor analysis tends to reveal larger correlations between
factors
 Effort and Importance at r = .47 vs. component analysis r = .38.

43
Implications of Test Dimensionality
 Dimensionality of test scores has implications for how test should be
scored.
 Types of Scores in Multidimensional Tests
 Subtest Score: Reflects performance on items within a single
dimension.
 Area Score: Aggregates several related subtests, offering a broader
view than a subtest score but not covering all test dimensions.
 Total Score: Represents comprehensive performance across all
dimensions of the test.
 Each score type (subtest, area, total) provides a unique
interpretative angle

44
Intelligence Test Score Levels
 Total IQ Score
 Represents an overall measure of intellectual capacity,
aggregating various cognitive domains.

 Area Scores
 Reflect specific domains of intelligence, such as 'Verbal
Intelligence' for language tasks and 'Spatial Intelligence' for
visual-spatial skills.

 Subtest Scores
 Measure discrete cognitive functions, e.g., 'Vocabulary' for
word knowledge, 'Block Design' for spatial pattern
reconstruction, and 'Comprehension' for language
understanding.

 The integration of subtest and area scores contributes

to the total IQ score, providing a detailed and holistic
view of an individual's cognitive abilities.
45
Ensuring Reliability and Validity
 Psychometric analyses needed at all scoring levels
 subtest, area, and total
 Each subtest score must reliably measure specific constructs
 Verify that groupings of subtest scores accurately represent larger
(Area) cognitive domains.
 Overall test score evaluated for its comprehensive reliability and
validity in measuring the construct.


46
Uncorrelated Dimensions: NEO PI-R
 Personality type assessments often feature uncorrelated dimensions.
 NEO-PI-R is an Example
 Measures the Big Five personality traits (Neuroticism, Extraversion, Openness,
Agreeableness, Conscientiousness) with each scale designed to function
independently.
 Lack of significant overlap or correlation among scales, reflecting the Big Five
framework's conceptualization of personality as separate constructs.
 The uncorrelated structure underscores the theory that key personality domains
operate as distinct entities.

47
Uncorrelated Dimensions: NEO PI-R
 Below is a visual depiction of the NEO PI-R as a factor model.

 Allows for unique trait measurement, with individuals potentially scoring high on
one dimension and low on another.
 Examine traits in isolation, showcasing human personality diversity.
 Inappropriate to aggregate a total score from the NEO PI-R due to the orthogonal
structure of personality dimensions.
 Data reduction results inform the calculation and interpretation of composite
scores
48
Sample Size and Data Reduction
 Dependent on variables' communality and the number of variables per factor.
 Importance of Communality
 Higher shared variance among variables can lower the necessary sample size for reliable
analysis.

 Variables Per Factor

 More variables per factor enhance factor estimation accuracy, potentially reducing needed
sample size.

 Sample Size Guidelines

 For analyses with ≥5 variables per factor and communalities >.20, a sample size of around 150 is
considered sufficient.

 With respect to the SOS, a sample of 219 was used, with each factor defined by 5
variables having communalities above .20, thus, surpassing the adequate sample
size threshold.

49