CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT

CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT (2) some test administers don’t even have
to be present
TESTING AND ASSESSMENT (a) usually administered to larger
 Roots can be found in early twentieth century in France 1905 groups
 Alfred Binet published a test designed to help place Paris school
(b) test takers complete tasks
children
 WW1, military used the test to screen large numbers of recruits independently
quickly for intellectual and emotional problems b) Scoring and interpretation procedures
 WW2, military depend more on tests to screen recruits for service (1) score: a code or summary statement,
usually (but not necessarily) numerical in
PSYCHOLOGICAL
PSYCHOLOGICAL TESTING nature, that reflects an evaluation of
ASSESSMENT
Process of measuring performance on a test, task, interview, or
Gathering & integration of
psychology-related some other sample of behavior
psychology-related data for
variables by means of (2) scoring: process of assigning such
DEFINITION the purpose of making a
devices/procedures evaluative codes/ statements to
psychological evaluation with
designed to obtain a
accompany of tools. performance on tests, tasks, interviews,
sample of behavior
or other behavior samples.
To answer a referral question,
To obtain some gauge, (3) different types of score:
solve problem or arrive at a
OBJECTIVE usually numerical in (a) cut score: reference point,
decision thru the use of tools
nature
of evaluation usually numerical, derived by
Testing may be judgement and used to divide
PROCESS Typically individualized individualized or group
a set of data into two or
Key in the process of selecting Tester is not key into the
ROLE OF more classifications.
tests as well as in drawing process; may be
EVALUATOR (i) sometimes reached
conclusions substituted
SKILL OF Typically requires an educated Requires technician-like without any formal
EVALUATIOR selection, skill in evaluation skills method: in order to
Entail logical problem-solving “eyeball”, teachers
Typically yields a test
OUTCOME approach to answer the who decide what is
score
referral ques.
passing and what is
3 FORMS OF ASSESSMENT: failing.
1. COLLABORATIVE PSYCHOLOGICAL ASSESSMENT – assessor and (4) who scores it
assesse work as partners from initial contact through final feedback (a) self-scored by testtaker
2. THERAPEUTIC PSYCHOLOGICAL ASSESSMENT – self-discovery and
(b) computer
new understandings are encouraged throughout the assessment
process (c) trained examiner
3. DYNAMIC PSYCHOLOGICAL ASSESSMENT – follows a model (a) c) psychometric soundness/ technical quality
evaluation (b) intervention (a) evaluation. Provide a means for (1) psychometrics:the science of
evaluating how the assesse processes or benefits from some type of psychological measurement.
intervention during the course of evaluation. (a) referring to to how
consistently and how
Tools of Psychological Assessment
accurately a psychological test
A. The Test (a measuring device or procedure)
measures what it purports to
1. psychological test: a device or procedure designed to
measure.
measure variables related to psychology (intelligence,
(2) utility: refers to the usefulness or
personality, aptitude, interests, attitudes, or values)
practical value that a test or other tool
2. format: refers to the form, plan, structure, arrangement, and
of assessment has for a particular
layout of test items as well as to related considerations such as
purpose.
time limits.
B. The Interview: method of gathering information through direct
a) also referred to as the form in which a test is
communication involving reciprocal exchange
administered (pen and paper, computer,
1. interviewer in face-to-face is taking note of
etc) Computers can generate scenarios.
a) verbal language
b) term is also used to denote the form or structure of
b) nonverbal language
other evaluative tools, and processes, such as the
(1) body language movements
guidelines for creating a portfolio work sample
(2) facial expressions in response to
3. Ways That tests differ from one another:
interviewer
a) administrative procedures
(3) the extent of eye contact
(1) some test administers have an active
(4) apparent willingness to cooperate
knowledge
c) how they are dressed
(a) some test administration
(1) neat vs sloppy vs inappropriate
involves demonstration of
2. interviewer over the phone taking note of
tasks
a) changes in the interviewee’s voice pitch
(b) usually one-on-one
b) long pauses
(c) trained observation of
c) signs of emotion in response
assessee’s performance
3. ways that interviews differ:
a) length, purpose, and nature
 CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT
b) in order to help make diagnostic, treatment, 6. interpretive report: a formal or official computer-generated
selection, etc account of test performance presented in both numeric and
4. panel interview narrative form and including an explanation of the findings;
a) an interview conducted with one interviewee a) the three varieties of interpretive report are
with more than one interviewer (1) descriptive
C. The Portfolio (2) screening
1. files of work products: paper, canvas, film, video, audio, etc (3) consultive
2. samples of ones abilities and accomplishments b) some contain relatively little interpretation and
D. Case History Data: records, transcripts, and other accounts in written, simply call attention to certain high, low, or unusual
pictorial or other form that preserve archival information, official and scores that needed to be focused on.
informal accounts, and other data and items relevant to assessee c) consultative report: A type of interpretive report
1. sheds light on an individual's past and current adjustment as designed to provide expert and detailed analysis of
well as on events and circumstances that may have test data that mimics the work of an expert
contributed to any changes in adjustment consultant.
2. provides information about neuropsychological functioning d) integrative report: a form of interpretive report of
prior to the occurrence of a trauma or other event that psychological assessment, usually computer-
results in a deficit. generated, in which data from behavioral, medical,
3. insight into current academic and behavioral standing administrative, and/or other sources are
4. useful in making judgments for future class placements integrated
5. Case history Study: a report or illustrative account 7. CAPA: computer assisted psychological assessment. (assistance
concerning person or an event that was compiled on the basis to the test user not the test taker)
of case history data a) enables test developers to create psychometrically
a) might shed light on how one individual’s personality sound tests using complex mathematical procedures
and particular set of environmental conditions and calculations.
combined to produce a successful world leader. b) enables test users the construction of tailor-made
b) groupthink: work on a social psychological test with built-in scoring and interpretive capabilities.
phenomenon: contains rich case history material on c) Pros:
collective decision making that did not always result (1) test administrators have greater access to
in the best decisions. potential test users because of the global
E. Behavioral Observation: monitoring the actions of others or oneself by reach of the internet.
visual or electronic means while recording quantitative and/or qualitative (2) scoring and interpretation of test data
information regarding those actions. tend to be quicker than for paper-and-
1. often used as a diagnostic aid in various settings: inpatient pencil tests
facilities, behavioral research laboratories, classrooms. (3) costs associated with internet testing tend
2. naturalistic observation: behavioral observation that takes to be lower than costs associated with
place in a naturally occurring setting (as opposed to a research paper-and-pencil tests
laboratory) for the purpose of evaluation and information- (4) the internet facilitates the testing of
gathering. otherwise isolated populations, as well as
3. in practice tends to be used most frequently by researchers people with disabilities for whom getting
in settings such as classrooms, clinics, prisons, etc. to a test center might prove as a hardship.
F. Role- Play Tests (5) greener: conserves paper, shipping
1. role play: acting an improvised or partially improvised part in a materials etc.
simulated situation. d) Cons:
2. role-play test: tool of assessment wherein assessees are (1) test client integrity
directed to act as if they were in a particular situation. Assessees (a) refers to the verification of the
are then evaluated with regard to their expressed thoughts, identity of the test taker when
behaviors, abilities, etc a test is administered online
G. Computers as tools (b) also refers to the sometimes
1. local processing: on site computerized scoring, interpretation, varying interests of the test
or other conversion of raw test data; contrast w/ CP and taker vs that of the test
teleprocessing administrator. The test taker
2. central processing: computerized scoring, interpretation, or might have access to notes,
other conversion of raw data that is physically transported from aids, internet resources etc.
the same or other test sites; contrast w/ LP and teleprocessing. (c) internet testing is only testing,
3. teleprocessing: computerized scoring, interpretation, or other not assessment
conversion of raw test data sent over telephone lines by modem 8. CAT: computerized adaptive testing: an interactive, computer-
from a test site to a central location for computer processing. administered test taking process wherein items presented to
contrast with CP and LP the test taker are based in part on the test taker's
4. simple score report: a type of scoring report that provides only performance on previous items
a listing of scores a) EX: on a computerized test of academic abilities, the
5. extended scoring report: a type of scoring report that computer might be programmed to switch from
provides a listing of scores AND statistical data. testing math skills to English skills after three
consecutive failures on math items.
H. Other Tools
 CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT
1. DVD- how would you respond to the events that take place in satisfaction, personal values, quality of living conditions,
the video and quality of friendships and other social support.
a) sexual harassment in the workplace  BUSINESS AND MILITARY SETTINGS
b) respond to various types of emergencies  GOVERNMENTAL AND ORGANIZATIONAL CREDENTIALING
c) diagnosis/treatment plan for clients on videotape How are Assessments Conducted?
2. thermometers, biofeedback, etc  protocol: the form or sheet or booklet on which a testtaker’s
responses are entered.
TEST DEVELOPER o term might also be used to refer to a description of a set of
 They are the one who create tests. test- or assessment- related procedures, as in the
 They conceive, prepare, and develop tests. They also find a way to sentence , “the examiner dutifully followed the complete
disseminate their tests, by publishing them either commercially or protocol for the stress interview”
through professional publications such as books or periodicals.  rapport: working relationship between the examiner and the
TEST USER
examinee
 They select or decide to take a specific test off the shelf and use it for
some purpose. They may also participate in other roles, e.g., as
examiners or scorers. ASSESSEMENT OF PEOPLE WITH DISABILITITES
TEST TAKER  Define who requires alternate assessement, how such assessment are
 Anyone who is the subject of an assessment to be conducted and how meaningful inferences are to be drawn
 Test taker may vary on a continuum with respect to from the data derived from such assessment
numerous variables including:  Accommodation – adaptation of a test, procedure or situation or the
o The amount of anxiety they experience & the degree to substitution of one test for another to make the assessment more
which the test anxiety might affect the results suitable for an assesee with exceptional needs.
o The extent to which they understand & agree with  Translate it into Braillee and administere in that form.
the rationale of the assessment  Alternate assessment – evaluative or diagnostic procedure or process
o Their capacity & willingness to cooperate that varies from the usual, customary, or standardized way a
o Amount of physical pain/emotional distress they are measurement is derived either by virtue of some special
experiencing accommodation made to the assesee by means of alternative
o Amount of physical discomfort methods
o Extent to which they are alert & wide awake  Consider these four variables on which of many different types of
o Extent to which they are predisposed to agreeing accommodation should be employed:
or disagreeing when presented with stimulus o The capabilities of the assesse
o The extent to which they have received prior coaching o The purpose of the assessment
o May attribute to portraying themselves in a good light o The meaning attached to test scores
 Psychological autopsy – reconstruction of a deceased individual’s o The capabilities of the assessor
psychological profile on the basis of archival records, artifacts, & REFERENCE SOURCES
interviews previously conducted with the deceased assesee  TEST CATALOUGES – contains brief description of the test
TYPES OF SETTINGS  TEST MANUALS – detailed information
 EDUCATIONAL SETTING  REFERENCE VOLUMES – one stop shopping, provides detailed
o achievement test: evaluation of accomplishments or the information for each test listed, including test publisher,
degree of learning that has taken place, usually with author, purpose, intended test population and test
administration time
regard to an academic area.
 JOURNAL ARTICLES – contain reviews of the test
o diagnosis: a description or conclusion reached on the basis
 ONLINE DATABASES – most widely used bibliographic databases
of evidence and opinion though a process of distinguishing
the nature of something and ruling out alternative TYPES OF TESTS
conclusions.  INDIVIDUAL TEST – those given to only one person at a time
o diagnostic test: a tool used to make a diagnosis, usually to  GROUP TEST – administered to more than one person at a time by
identify areas of deficit to be targeted for intervention single examiner
o informal evaluation: A typically non systematic, relatively  ABILITY TESTS:
o ACHIEVEMENT TESTS – refers to previous learning (ex.
brief, and “off the record” assessment leading to the
Spelling)
formation of an opinion or attitude, conducted by any
o APTITUDE/PROGNOSTIC – refers to the potential for
person in any way for any reason, in an unofficial context learning or acquiring a specific skill
and not subject to the same ethics or standards as o INTELLIGENCE TESTS – refers to a person’s general
evaluation by a professiomal potential to solve problems
 CLINICAL SETTING  PERSONALITY TESTS: refers to overt and covert dispositions
o these tools are used to help screen for or o OBJECTIVE/STRUCTURED TESTS – usually self-report,
diagnose behavior problems require the subject to choose between two or more
o group testing is used primarily for screening: identifying alternative responses
those individuals who require further diagnostic o PROJECTIVE/UNSTRUCTURED TESTS – refers to all
evaluation. possible uses, applications and underlying concepts of
 COUNSELING SETTING psychological and educational tests
o schools,prisons, and governmental or privately owned o INTEREST TESTS –
institutions
o ultimate objective: the improvement of the assessee in
terms of adjustment, productivity, or some related
variable.
 GERIATRIC SETTING
o quality of life: in psychological assesment, an evaluation
of variables such as perceived stress,lonliness, sources of
 CHAPTER 2: HISTORICAL, CULTURAL AND LEGAL/ETHICAL
A HISTORICAL PERSPECTIVE testakers from young children through senior
19TH CENTURY adulthood.
 Tests and testing programs first came into being in China B. THE MEASUREMENT OF PERSONALITY
 Testing was instituted as a means of selecting who, of many o Field of psychology was being too test oriented
applicants would obtain government jobs (Civil service) o Clinical psychology was synonymous to mental testing
 The job applicants are tested on proficiency in endeavors such o ROBERT WOODWORTH – develop a measure of
as music, archery, knowledge and skill etc. adjustment and emotional stability that could be
GRECO-ROMAN WRITINGS (Middle Ages) administered quickly and efficiently to groups of recruits
 World of evilness  To disguise the true purpose of the test,
 Deficiency in some bodily fluid as a factor believed to influence questionnaire was labeled as Personal Data
personality Sheet
 Hippocrates and Galen  He called it Woodworth Psychoneurotic
RENAISSANCE Inventory – first widely used self-report test of
 Christian von Wolff – anticipated psychology as a science and personality
psychological measurement as a specialty within that science o Self-report test:
CHARLES DARWIN AND INDIVIDUAL DIFFERENCES  Advantages:
 Tests designed to measure these individual differences in ability and  Respondents best qualified
personality among people  Disadvantages:
 “Origin of Species” chance variation in species would be selected  Poor insight into self
or rejected by nature according to adaptivity and survival value.  One might honestly believe
“survival of the fittest” something about self that isn’t true
FRANCIS GALTON  Unwillingness to report seemingly
 Explore and quantify individual differences between people. negative qualities
 Classify people “according to their natural gifts” o Projective test: individual is assumed to project onto some
 Displayed the first anthropometric laboratory ambiguous stimulus (inkblot, photo, etc.) his or her own
KARL PEARSON unique needs, fears, hopes, and motivations
 Developed the product moment correlation technique.  Ex.) Rorschack inkblot
 His work can be traced directly from Galton o
WILHEM MAX WUNDT C. THE ACADEMIC AND APPLIED TRADITIONS
 First experimental psychology laboratory in University of Leipzig
Culture and Assessment
 Focuses more on relating to how people were similar, not different
from each other.
Culture: ‘the socially transmitted behavior patterns, beliefs, and products of
JAMES MCKEEN CATELL
work f a particular population, community, or group of people’
 Individual differences in reaction time
 Coined the term mental test
Evolving Interest in Culture-Related Issues
CHARLES SPEARMAN
Goddard tested immigrants and found most to be feebleminded
 Originating the concept of test reliability as well as building the
-invalid; overestimated mental deficiency, even in native English-
mathematical framework for the statistical technique of factor
speakers
analysis
Lead to nature-nurture debate about what intelligence tests actually measure
VICTOR HENRI
Needed to “isolate” the cultural variable
 Frenchman who collaborated with Binet on papers suggesting how Culture-specific tests: tests designed for use with ppl from one culture, but not
mental tests could be used to measure higher mental processes from another
EMIL KRAEPELIN -minorities still scored abnormally low
 Early experimenter of word association technique as a formal test ex.) loaf of bread vs. tortillas
LIGHTNER WITMER today tests undergo many steps to ensure its suitable for said nation
 “Little known founder of clinical psychology” -take testtakers reactions into account
 Founded the first psychological clinic in the U.S.
PSYCHE CATELL Some Issues Regarding Culture and Assessment
 Daughter of James Cattell  Verbal Communication
 Cattel Infant Intelligence Scale (CIIS) & Measurement of Intelligence in o Examiner and examinee must speak the same language
Infants and Young Children o Especially tricky with infrequently used vocabulary or
RAYMOND CATTELL unusual idioms employed
 Believed in lexical approach to defining personality which examines o Translator may lose nuances of translation or give
human languages for descriptors of personality dimensions unintentional hints toward more desirable answer
20th CENTURY o Also requires understanding of culture
- Birth of the first formal tests of intelligence  Nonverbal Communication and Behavior
- Testing shifted to be of more understandable relevance/meaning o Different between cultures
A. THE MEASUREMENT OF INTELLIGENCE o Ex.) meaning of not making eye contact
o Binet created first intelligence to test to identify mentally o Body movement could even have physical cause
retarded school children in Paris (individual) o Psychoanalysis: Freud’s theory of personality and
o Binet-Simon Test has been revised over again psychological treatment which stated that symbolic
o Group intelligence tests emerged with need to significance is assigned to many nonverbal acts.
screen intellect of WWI recruits o Timing tests in cultures not obsessed with speed
o David Wechsler – designed a test to measure adult o Lack of speaking could be reverence for elders
intelligence test  Standards of Evaluation
 for him Intelligence is a global capacity of the o Acceptable roles for women differ throughout culture
individual to act purposefully, to think o “judgments as to who might be the best employee,
rationally and to deal effectively with his manager, or leader may differ as a function of culture, as
environment. might judgments regarding intelligence, wisdom, courage,
 Wechsler-Bellevue Intelligence Scale  and other psychological variables”
Wechsler Adult Intelligence Test – was revised
several times and extended the age range of
 CHAPTER 2: HISTORICAL, CULTURAL AND LEGAL/ETHICAL
o must ask ‘how appropriate are the norms or other  The right to be informed of test findings
standards that will be used to make this evaluation’ o Formerly test administrators told to give participants only
positive information
Tests and Group Membership o No realistic information is required
 ex.) must be 5’4” to be police officer- excludes cultures with short o Tell test takers as little as possible about the nature of
stature their performance on a particular test. So that the
 ex.) Jewish lifestyle not well suited for corporate America examinee would leave the test session feeling pleased and
 affirmative action: voluntary and mandatory efforts to combat statisfied.
discrimination and promote equal opportunity in education and o Test takers have the right also to know what
employment for all recommendations are being made as a consequence of the
 Psychology, tests, and public policy test data
 The right to privacy and confidentiality
Legal and Ethical Condiseration o Private right: “recognizes the freedom of the individual to
Code of professional ethics: defines the standard of care expected of members of pick and choose for himself the time, circumstances, and
a given profession. particularly the extent to which he wishes to share or
withhold from others his attitudes, beliefs, behaviors, and
The Concerns of the Public opinions”
 Beginning in world war I, fear that tests were only testing the ability o Privileged information: information protected by law from
to take tests being disclosed in legal proceeding. Protects clients from
 Legislation disclosure in judicial proceedings. Privilege belongs to the
o Minimum competency testing programs: formal testing client not the psychologist.
programs designed to be used in decisions regarding o Confidentiality: concerns matters of communication
various aspects of students’ educations outside the courtroom
o Truth-in-testing legislation: state laws to provide testtakers  Safekeeping of test data: It is not a good policy
with a means of learning the criteria by which they are to maintain all records in perpetuity
being judged  The right to the least stigmatizing label
 Litigation o The standards advise that the least stigmatizing labels
o Daubert ruling made federal judges the gatekeepers to should always be assigned when reporting test results.
determining what expert testimony is admitted
o This overrode the Frye policy which only admitted
scientific testimony that had won general acceptance in
the scientific community.

The Concerns of the Profession

 Test-user qualifications
o Who should be allowed to use psych tests
o Level A: tests or aids that can adequately be administered,
scored, and interpreted with the aid of the manual and a
general orientation to the kind of institution or
organization in which one is working
o Level B: tests or aids that require some technical
knowledge of test construction and use and of supporting
psychological and educational fields
o Level C: tests and aids requiring substantial understanding
of testing and supporting psych fields with experience
 Testing people with disabilities
o Difficulty in transforming the test into a form that can be
taken by testtaker
o Transferring responses to be scorable
o Meaningfully interpreting the test data
 Computerized test administration, scoring, and interpretation
o simple, convenient
o easily copied, duplicated
o insufficient research to compare it to pencil-and-paper
versions
o value of computer interpretation is questionable
o unprofessional, unregulated “psychological testing” online

The Rights of Testtakers

 the right of informed consent
o right to know why they are being evaluated, how test data
will be used and what information will be released to
whom
o may be obtained by parent or legal representative
o must be in written form:
 general purpose of the testing
 the specific reason it is being undertaken
 general type of instruments to be administered
o revealing this information before the test can contaminate
the results
o deception only used if absolutely necessary
o don’t use deception if it will cause emotional distress
o fully debrief participants
 CHAPTER 3: A STATISTICS
 No absolute zero point
Why We Need Statistics  Can take average
RATIO SCALE
- Statistics are important for purposes of education  In addition to all the properties of nominal, ordinal, and interval
o Numbers provide convenient summaries and allow us to measurement, ratio scale has true zero point
evaluate some observations relative to others  Equal intervals between numbers
- We use statistics to make inferences, which are logical deductions  Ex.) measuring amount of pressure hand can exert
about events that cannot be observed directly  True zero doesn’t mean someone will receive a score of 0, but means
o Detective work of gathering and displaying clues – that 0 has meaning
exploratory data analysis
o Then confirmatory data analysis NOTE:
- Descriptive statistics are methods used to provide a concise Permissible Operations
description of a collection of quantitative information - Level of measurement is important because it defines which
- Inferential statistics are methods used to make inferences from mathematical operations we can apply to numerical data
observations of a small group of people known as a sample to a larger - For nominal data, each observation can be placed in only one
group of individuals known as a population mutually exclusive category
- Ordinal measurements can be manipulated using arithmetic
SCALES OF MEASUREMENT - With interval data, one can apply any arithmetic operation to the
differences between scores
 MEASUREMENT – act of assigning numbers or symbols to o Cannot be used to make statements about ratios
characteristics of things according to rules. The rules serves as a
guideline for representing the magnitude. It always involves error. DESCRIBING DATA
 SCALE – set of numbers whose properties model empirical properties  Distribution: set of scores arrayed for recording or study
of the objects to which the numbers are assigned.  Raw Score: straightforward, unmodified accounting of performance,
 CONTINUOUS SCALE – interval/ratio. A scale used to measure usually numerical
continuous variable. Always involves error
 DISCRETE SCALE – nominal/ordinal used to measure a discrete Frequency Distributions
variable (ex. Female or male)  Frequency Distribution: All scores listed alongside the number of
 ERROR – collective influence of all of the factors on a test score. times each score occurred
 Grouped Frequency Distribution: test-score intervals (class intervals),
PROPERTIES OF SCALES replace the actual test scores
- Magnitude, equal intervals, and an absolute 0
o Highest and lowest class intervals= upper and lower limits
Magnitude of distribution
- The property of “moreness”
 Histogram: graph with vertical lines drawn at the true limits of each
- A scale has the property of magnitude if we can say that a particular
test score (or class interval) forming TOUCHING rectangles- midpoint
instance of the attribute represents more, less, or equal amounts of in center of bar
the given quantity than does another instance
 Bar Graph: rectangles DON’T touch
Equal Intervals
 Frequency Polygon: data illustrated with continuous line connecting
- A scale has the property of equal intervals if the difference between
the points where test scores or class intervals meet frequencies
two points at any place on the scale has the same meaning as the
 A single test score means more if one relates it to other test scores
difference between two other points that differ by the same number
 A distribution of scores summarizes the scores for a group of
of scale units
individuals
- A psychological test rarely has the property of equal intervals
 Frequency distribution: displays scores on a variable or a measure to
- When a scale has the property of equal intervals, the relationship
reflect how frequently each value was obtained
between the measured units and some outcome can be described by
a straight line or a linear equation in the form Y=a+bX o One defines all the possible scores and determines how
many people obtained each of those scores
o Shows that an increase in equal units on a given scale
reflects equal increases in the meaningful correlates of  Income is an example of a variable that has a positive skew
units  Whenever you draw a frequency distribution or a frequency polygon,
Absolute 0 you must decide on the width of the class interval
- An Absolute 0 is obtained when nothing of the property being  Class interval: for inches of rainfall is the unit on the horizontal axis
measured exists
- This is extremely difficult/impossible for many psychological qualities Measures of Central Tendency
 Measure of central tendency: statistic that indicates the average or
NOMINAL SCALE midmost score between the extreme scores in a distribution.
 Simplest form of measurement  The Arithmetic Mean
 Classification or categorization o “X bar”
 Arithmetic operations can be performed with nominal data o sum of observations divided by number of observations
 Ex.) Male or female o Sigma (X/n)
 Also includes test items o Used for interval or ratio data when distributions are
relatively normal
o Ex.) yes/no responses
ORDINAL SCALE  The Median
o The middle score
 Classifies in some kind of ranking order
o Used for ordinal, interval, and ratio data
 Individuals compared to others and assigned a rank
o Especially useful when few scores fall at extremes
 Imply nothing about how much greater one ranking is than another
 The Mode
 Numbers/ranks do not indicate units of measure
o Most frequently-occurring score
 No absolute zero point
o Bimodal distribution- 2 scores both have highest
 Binet: believed that data derived from intelligence test are ordinal in
frequency
nature
o Only common with nominal data
INTERVAL SCALE
Measures of Variability
 In addition to the features of nominal and ordinal scales, contain
equal intervals between numbers
 CHAPTER 3: A STATISTICS
 Variability: indication of how scores in a distribution are scattered or  conversion of a raw score into a number indicating how many
dispersed standard deviation units the raw score is below or above the mean of
 The Range the distribution.
o Difference between the highest and lowest scores  The difference between a particular raw score and the mean divided
o Quick but gross description of the spread of scores by the standard deviation
 The interquartile and semi-interquartile range  Used to compare test scores with difference scales
o Distribution is split up by 3 quartiles, thus making 4
quarters each representing 25% of the scores T-score
o Q2= median  Standard score system composed of a scale that ranges from 5
o Interquartile range measure of variability equal to the standard deviations below the mean to 5 standard deviations above
difference between Q3 and Q1 the mean
o Semi-interquartile range interquartile range divided by 2  No negatives
 Quartiles and Deciles
o Quartiles are points that divide the frequency distribution Other Standard Scores
into equal fourths  SAT
o First quartile is the 25th percentile; second quartile is the  GRE
median, or 50th percentile; third quartile is the 75th  Linear transformation: when a standard score retains a direct
percentile numerical relationship to the original raw score
o The interquartile range is bounded by the range of scores  Nonlinear transformation: required when data are not normally
that represents the middle 50% of the distribution distributed, yet comparisons with normal distributions need to be
o Deciles are similar but use points that mark 10% rather made
than 25% intervals o Normalized Standard Scores
o Stanine system: converts any set of scores into  When scores don’t fall on normal distribution
a transformed scale, which ranges from 1 to 9  “normalizing a distribution involves ‘stretching’
 The average deviation he skewed curve into the shape of a normal
o X-mean=x curve and creating a corresponding scale of
o Average deviation= (sum of all deviation scores)/ total standard scores, a scale called a normalized
number of scores standard score scale”
o Tells us on average how far scores are from the mean
 The Standard Deviation
o Similar to average deviation
o But in order to overcome the (+/-) problem, each
deviation is squared
o Standard deviation: a measure of variability equal to the
square root of the average squared deviations about the
mean
o Is square root of variance
o Variance: the mean of the squares of the difference b/w
the scores in a distribution and their mean
 Found by squaring and summing all the
deviation scores and then dividing by the total
number of scores
o s = sample standard deviation
o sigma = population standard deviation
Skewness
 skewness: nature and extent to which symmetry is absent
 POSITIVE SKEW Ex.) test was too hard
 NEGATIVELY SKEWED ex.) test was too easy
 can be gauges by examining relative distances of quartiles from the
median
Kurtosis
 steepness of distribution
 platykurtic: relatively flat
 leptokurtic: relatively peaked
 mesokurtic: somewhere in the middle

The Normal Curve

Normal curve: bell-shaped, smooth, mathematically defined curve, highest at
center; both sides taper as it approaches the x-axis asymptotically
-symmetrical, and thus have mean, median, mode, is same

Area under the Normal Curve

Tails and body

Standard Scores
Standard Score: raw score that has been converted from one scale to another
scale, where the latter has arbitrarily set mean and standard deviation
-used for comparison

Z-score
 CHAPTER 4: OF TESTS AND
 Tasks on some tests mimic the actual behaviors that
Some Assumptions About Psychological Testing and Assessment the test user is attempting to understand
- Assumption 1: Psychological Traits and States Exist o Obtained behavior is usually used to predict future behavior
o Trait: any distinguishable, relatively enduring way in which one o Could also be used to postdict behavior to aid in the
individual varies from another understanding of behavior that has already taken place
o States: distinguish one person from another but are relatively o Tools of assessment, such as a diary, or case history data, might
less enduring be of great value in such an evaluation
 Trait term that an observer applies, as well as - Assumption 4: Tests and Other Measurement Techniques Have Strengths
strength or magnitude of the trait presumed present and Weaknesses
 based on observing a sample of behavior o Competent test users understand a lot about the tests they use
o Trait and state definitions also refer to individual variation  How it was developed
make comparisons with respect to the hypothetical average  Circumstances under which it is appropriate to
person administer the test
o Samples of behavior:  How test should be administered and to whom
 Direct observation  How results should be interpreted
 Analysis of self-report statements o Understand and appreciation limitations for tests they use
 Paper-and-pencil test answers - Assumption 5: Various Sources of Error Are Part of the Assessment Process
o Psychological trait  covers wide range of o Everyday error= misstates and miscalculations
possible characteristics; ex: o Assessment error= a long-standing assumption that
 Intelligence factors other than what a test attempts to measure will
 Specific intellectual abilities influence performance on a test
 Cognitive style o Error variance: component of a test score attributable to
 Psychopathology sources other than the trait or ability measured
o Controversy regarding how psychological tests exist  Assessees themselves are sources of error variance
 Psychological tests exist only as constructs: an o Classical test theory (CTT)/ True score theory: assumption is
informed, scientific concept developed or made that each testtaker has a true score on a test that would
constructed to describe or explain a behavior be obtained but for the action of measurement error
 Cant see, hear or touch infer - Assumption 6: Testing and Assessment Can Be Conducted in a Fair and
existence from overt behavior: refers to Unbiased Manner
an observable action or the product of o Court challenged to various tests and testing programs have
an observable action, including test- or sensitized test developers and users to the societal demand for
assessment-related responses fair tests used in a fair manner
o Traits not expected to be manifested in behavior 100% of the  Publishers strive to develop instruments that are fair
time when used in strict accordance with guidelines in the
 Seems to be rank-order stability in personality test manual
traits relatively high correlations between trait o Fairness related problems/questions:
scores at different time points  Culture is different from people whom the test was
o Whether and to what degree a trait manifests itself is intended for
dependent on the strength and nature of the situation  Politics
- Assumption 2: Psychological Traits and States Can Be Quantified and - Assumption 7: Testing and Assessment Benefit Society
Measured o Many critical decisions are based on testing and assessment
o After acknowledged that psychological traits and states do exist, procedures
the specific traits and states to be measured need to be defined
 What types of behaviors are assumed to WHAT’S A “GOOD TEST”?
be indicative of trait? - Criteria
 Test developer has to provide test users with a clear o Clear instruction for administration, scoring, and interpretation
operational definition of the construct under study - Reliability
o After being defined, test developer considers types of item o A “good test”/measuring tool reliable
content that would provide insight into it  Involves consistency: the prevision with which the
 Ex: behaviors that are indicative of a particular trait test measures and the extent to which error is
o Should all questions be weighted the same? present in measurements
 Weighting the comparative value of a test’s items  Unreliable measurement needs to be avoided
comes about as the result of a complex interplay - Validity
among many factors: o Test is considered valid if it doesn’t indeed measure what it
 Technical considerations purports to measure
 The way a construct has been defined (for o If there is controversy over the definition of a construct then the
particular test) validity is sure to be criticized as well
 Value society (and test developer) attach o Questions regarding validity focus on the items that collectively
to behaviors evaluated make up the test
o Need to find appropriate ways to score the test and interpret  Adequately sample range of areas to measure
results construct
 Cumulative scoring: test score is presumed to  Individual items contribute to or take away from
represent the strength of the targeted ability or test’s validity
trait or state o Validity may also be questioned on grounds related to the
 The more the testtaker responds in a interpretation of test results
particular direction (as keyed by test - Other Considerations
manual) the higher the testtaker is o “Good test” one that trained examiners can administer,
presumed to possess the targeted trait score and interpret with minimum difficulty
or ability  Useful
- Assumption 3: Test-Related Behavior Predicts Non-Test-Related Behavior  Yields actionable results that will ultimately benefit
o Objective of test is to provide some indication of some aspects individual testtakers or society at large
of the examinee’s behavior
 CHAPTER 4: OF TESTS AND
o Purpose of test compare performance of testtaker with o STANDARD ERROR OF THE DIFFERENCE – estimate how
performance of other testtakers (contains adequate norms: large a difference between two scores should be before
normative data) the difference is considered statistically significant
 Normative data provides standard with which results - Developing norms for a standardized test
measured can be compared o Establish a standard set of instructions and conditions
NORMS under which the test is given makes scores of normative
- Norm-referenced testing and assessment: method of evaluation and sample more comparable with scores of future testtakers
a way of deriving meaning from test scored by evaluating an o All data collected and analyzed, test developer will
individual testtaker’s score and comparing it to scores of a group of summarize data using descriptive statistics (measures of
testtakers central tendency and variability)
- Meaning of individual score is relative to other scores on the same  Test developer needs to provide precise
test description of standardization sample itself
- Norms (scholarly context): usual, average, normal, standard,  Descriptions of normative samples vary widely
expected or typical in detail
- Norms (psychometric context): the test performance data of a Tracking
particular group of testtakers that are designed for use as a reference - Comparisons are usually with people of the same age
when evaluating or interpreting individual test scores - Children at the same age level tend to go through different growth
- Normative sample: group of people whose performance on a patterns
particular test is analyzed for reference in evaluation the - Pediatricians must know the child’s percentile within a given age
performance of individual testtakers group
o Yields a distribution of scores - This tendency to stay at about the same level relative to one’s peers is
- Norming: refers to the process of deriving norms; particular type of known as tracking (ie height and weight)
norm derivation - Diets may alter this “track”
o Race norming: controversial practice of norming on the - Faults: some believe there is an analogy between the rates of physical
basis of race or ethnic background growth and the rates of intellectual growth
- Norming a test can be very expensive user norms/program norms: o Some say that children learn at different rates
consist of descriptive statistics based on a group of testtakers in a o This system discriminates against some children
given period of time rather than norms obtained by form sampling
methods TYPES OF NORMS
- Sampling to Develop Norms o Classification of norms ex: age, grade, national, local,
- Standardization: process of administering a test to a percentile, etc.
representative sample of testtakers for the purpose of establishing o PERCENTILES
norms  Median= 2nd quartile: the point at or below which
o Standardized when has clear, specified procedures 50% of the scores fell and above which the remaining
- Sampling 50% fell
o Developer targets defined group as population test  Might wish to divide distribution of scores into
designed for deciles (instead of quartiles): 10 equal parts
 All have at least one common, observable  The Xth percentile is equal to the score at or below
characteristic which X% of scores fall
o To obtain distribution of scores:  Percentile: an expression of the percentage of
 Test administered to everyone in people whose score on a test or measure falls below
targeted population a particular raw score
 Administer test to a sample of the population  Percentage correct: refers to the
 Sample: portion of universe of distribution of raw scores (number of
people deemed to be representative items that were answered correctly)
of whole population multiplied by 100 and divided by the total
 Sampling: process of selecting the number of items *not same as percentile
portion of universe deemed to be  Percentile is a converted score that refers
representative of whole to a percentage of testtakers
o Subgroups within a defined population may differ with  Percentiles are easily calculated popular way of
respect to some characteristics and it is sometimes organizing test related data
essential to have these differences proportionately  Using percentiles with normal distribution real
represented in sample differences between raw scores may be minimized
 Stratified sampling: sample reflects statistics of near the ends of the distribution and exaggerated in
whole population; helps prevent sampling bias the middle (worsens with highly skewed data)
and ultimately aid in interpretation of findings o AGE NORMS
 Purposive sampling: arbitrarily select sample  Age-equivalent scores/age norms: indicate the
we believe to be representative of population average performance of different samples of
 Incidental/convenience sampling: sample that testtakers who were at various ages at the time the
is convenient or available for use test was administered
 Very exclusive (contain exclusionary  Age norm tables for physical
criteria) characteristics
- TYPES OF STANDARD ERROR:  “Mental” age vs. physical age (need to
o STANDARD ERROR OF MEASUREMENT – estimate the identify mental age)
extent to which an observed score deviates from a true o GRADE NORMS
score  Grade norms: designed to indicate the average test
o STANDARD ERROR OF ESTIMATE – In regression, an performance of testtakers in a given school grade
estimate of the degree of error involved in predicting the  Developed by administering the test to
value of one variable from another representative samples of children over a
o STANDARD ERROR OF THE MEAN – a measure of sampling range of consecutive grades
error  Mean or median score for children at
each grade level is calculated
 CHAPTER 4: OF TESTS AND
 Great intuitive appeal CORRELATION
 Do not provide info as to the content or  Degree and direction of correspondence between two things.
type of items that a student could or  Correlation coefficient (r) – expresses a linear relationship between
could not answer correctly two continuous variables
 Developmental norms: (ex: grade norms and age o Numerical index that tells us the extent to which X and
norms) term applied broadly to norms developed on Y are “co-related”
the basis of any trait, ability, skill, or other  Positive correlation: high scores on Y are associated with high scores
characteristic that is presumed to develop, on X, and low scores on Y correspond to low scores on X
deteriorate, or otherwise be affected by  Negative correlation: higher scores on Y are associated with lower
chronological age, school grade, or stage of life scores on X, and vise versa
o NATIONAL NORMS  No correlation: the variables are not related
 National norms: derived from a normative sample  -1 to 1
that was nationally representative of the population  Correlation does not imply causation.
at the time the norming study was conducted o Ie weight, height, intelligence
o NATIONAL ANCHOR NORMS
 Many different tests purporting to measure the same PEARSON r
human characteristics or abilities  Pearson Product Moment Correlation Coefficient
 National anchor norms: equivalency tables for scores  Devised by Karl Pearson
on tests that purpose to measure the same thing  Relationship of two variables are linear and continuous
 Could provide the tool for comparisons  Coefficient of Determination (r2) – indication of how much variance is
 Provides stability to test scores by shared by the X and the Y variables
anchoring them to other test scores SPEARMAN RHO
 Begins with the computation of percentile  Rank order correlation coefficient
norms for each test to be compared  Developed by Charles Spearman
 Equipercentile method: equivalency of  Used when the sample size is small and when both sets of
scores on different tests is calculated with measurements are in ordinal form (ranking form)
reference to corresponding percentile BISERIAL CORRELATION
scores  expresses the relationship between a continuous variable and an
o SUBGROUP NORMS artificial dichotomous variable
 Normative sample can be segmented by an criteria o If the dichotomous variable had been true then we would
initially used in selecting subjects for sample use the point biserial correlation
 Subgroup norms: result of segmentation; more o When both variables are dichotomous and at least one
narrowly defined of the dichotomies is true, then the association between
o LOCAL NORMS them can be estimated using the phi coefficient
 Local norms: provide normative info with respect to o If both dichotomous variables are artificial, we might use a
the local population’s performance on some test special correlation coefficient – tetrachoric correlation
 Typically developed by test
users themselves REGRESSION
- Fixed Reference Group Scoring Systems  analysis of relationships among variables for the purpose of
o Norms provide context for interpreting meaning of a test score understanding how one variable may predict another
o Fixed reference group scoring system: distribution of scored  SIMPLE REGRESSION: one IV (X) and one DV (Y)
obtained on the test from one group of testtakers (fixed - Regression line: defined as the best-fitting straight line through a set
reference group) is used as the basis for the calculation of test of points in a scatter diagram
scores for future administrators on the test o Found by using the principle of least squares, which
 Ex: SAT test (developed in 1962) minimizes the squared deviation around the regression
NORM-REFERENCED VERSUS CRITERION-REFERENCED EVALUATION line
- Way to derive meaning from test score is to evaluate test score  Primary use: To predict one score or variable from another
in relation to other scores on same test (Norm-referenced)  Standard error of estimate: the higher the correlation between X and
- Criterion-referenced: derive meaning from a test score by Y, the greater the accuracy of the prediction and the smaller the SEE.
evaluating it on the basis of whether or not some criterion has been  MULTIPLE REGRESSION: The use of more than one score to predict Y.
met  Regression coefficient: (b) slope of the regression line
o Criterion: a standard on which a judgment or decision may
o Sum of squares for the covariance to the sum of squares
be based
for X
- Criterion-referenced testing and assessment: method of evaluation
o Sum of squares is defined as the sum of the squared
and way of deriving meaning from test scores by evaluating an
deviations around the mean
individual’s score with reference to a set standard (ex: to drive must
o Covariance is used to express how much two measures
past driving test) covary, or vary together
o Derives from values and standards of an individual or
 Slope describes how much change is expected in Y each time X
organization
increases by one unit
o Also called Domain/content-referenced testing and
 Intercept (a) is the value of Y when X is 0
assessment
o The point at which the regression line crosses the Y axis
o Critique: if followed strictly, important info about
THE BEST-FITTING LINE
individual’s performance relative to others can
 The difference between the observed and predicted score (Y-Y’) is
be potentially lost
called the residual
Culture and Inference
- Culture is a factor in test administration, scoring and interpretation  The best-fitting line is most appropriately found by squaring each
- Test user should do research in advance on test’s available norms to residual
check how appropriate it is for targeted testtaker population  Best-fitting line is obtained by keeping these squared residuals as
o Helpful to know about the culture of the testtaker small as possible
o Principle of least squares:
CORRELATION AND INFERENCE  Correlation is a special case of regression in which the scores for
both variables are in standardized, or Z, units
 CHAPTER 4: OF TESTS AND
 In correlation, the intercept is always 0 - Third variable, ie poor social adjustment, causes TV viewing and
 Pearson product moment correlation coefficient is a ratio used to aggression
determine the degree of variation in one variable that can be - External influence is the third variable
estimated from knowledge about variation in the other variable Restricted Range
Testing the Statistical Significance of a Correlation Coefficient - Correlation and regression use variability on one variable to
- Begin with the null hypothesis that there is no relationship between explain variability on a second variable
variables - Restricted range problem: correlation requires variability; if the
- Null hypothesis rejected is there is evidence that the association variability is restricted, then significant correlations are difficult to
between two variables is significantly different from 0 find
- t distribution is not a single distribution, but a family of distributions, Mulvariate Analysis
each with its own degrees of freedom - Multivariate analysis considers the relationship among combinations
- Degrees of freedom are defined as the sample size minus 2, or N-2 of three of more variables
- Two-tailed test General Approach
- Linear combination of variables is a weighted composite of the
How to Interpret a Regression Plot original variables
- Regression plots are pictures that show the relationship between - Y’ = a+b1X1 + … bkXk
variables
- Common use of correlation is to determine the criterion validity
evidence for a test, or the relationship between a test score and
some well-defined criterion
- Middle level of enjoyableness because it is the one observed most
frequently – normative because it uses info gained from
representative groups
- Using the test as a predictor is not as good as perfect prediction, but
it is still better than using the normative info
- A regression line such as in 3.9 shows that the test score tells us
nothing about the criterion beyond the normative info

TERMS AND ISSUES IN THE USE OF CORRELATION

Residual
- Difference between the predicted and the observed values is called
the residual
o Y-Y’
- Important property of residual is that the sum of the residuals always
equals 0
- Sum of the squared residuals is the smallest value according to the
principle of least squares
Standard Error of Estimate
- Standard deviation of the residuals is the standard error of estimate
- A measure of the accuracy of prediction
- Prediction is most accurate when the standard error of estimate is
relatively small
Coefficient of Determination
- Correlation coefficient squared is known as the coefficient of
determination
- Tells us the proportion of the total variation in scores on Y that we
know as a function of information about X
Coefficient of Alienation
- Coefficient of alienation is a measure of nonassociation between two
variables
- Square root of 1-r2 –-- r is the coefficient of determination
- High value means there is a high degree of nonassociation between 2
variables
Shrinkage
- Tendency to overestimate the relationship, particularly if the sample
of subjects is small
- Shrinkage is the amount of decrease observed when a regression
equation is created for one population and then applied to another
Cross Validation
- Use regression equation to predict performance in a group of
subjects other than the ones to which the equation was applied
- Standard error of estimate obtained for relationship between the
values predicted by the equation and the values actually observed –
called cross validation
The Correlation-Causation Problem
- Experiments are required to determine whether manipulation of one
variable causes changes in another variable
- A correlation alone does not prove causality, although it might lead to
other research that is designed to establish the causal relationships
between variables
Third Variable Explanation
 CHAPTER 5:
RELIABILITY - TEST CONSTUCTION
- Dependability and consistent o Item sampling or content sampling – refer to variation
- Error implies that there will always be some inaccuracy in our among items within a test as well as to variation among
measurements items between test\
- Tests that are relatively free of measurement error are deemed to be  The extent to which a test takers score is
reliable affected by the content sampled on a test and
- Reliability estimates in the range of .70 and .80 are good enough for by the way the content is sampled (that is, the
most purposes in basic research way in which the item is constructed) is a
- Reliability coefficient: an index that indicates the ratio between the source of error variance
true score variance on a test and the total variance - TEST ADMINISTRATION
- HISTORY OF RELIABILITY: o may influence the test takers attention or motivation
o Charles Spearman (1904): The Proof and Measurement o Environment variables, test taker’s variables, examiner
of Association between Two Things variables. Level of professionalism
o Then Thorndike - TEST SCORING AND INTERPRETATION
o Item response theory has taken advantage of computer o Computer scoring and a growing reliance on objective,
technology to advance psychological measurement computer-scorable items have virtually eliminated error
significantly variance caused by scorer differences
o Based on Spearman’s ideas o However, other tools of assessment still require scoring by
- X = T + E  CLASSICAL TEST THEORY trained personnel
o assumes that each person has a true score that would be o If subjectivity is involved in scoring, then the scorer can be
obtained if there were no errors in measurement a source of error variance
o Difference between the true score and the observed score o Despite rigorous scoring criteria set forth in many of the
results from measurement error better known test of intelligence, examiner occasionally
still are confronted by situations where an examinees
o Assumption here is that errors of measurement are
response lies in a gray area
random
o Basic sampling theory tells us that the distribution of
TEST-RETEST RELIABILITY
random errors is bell-shaped
- Also known as time-sampling reliability
 The center of the distribution should represent
- Correlating pairs of scores from the same group on two different
the true score, and the dispersion around the
administration of the same test
mean of the distribution should display the
- Measure something that is relatively stable over time
distribution of sampling errors
- Sources of Error variance:
o Classical test theory assumes that the true score for an
o Passage of time: the longer the time that passes, the
individual will not change with repeated applications of
greater the likelihood that reliability coefficient will be
the same test
lower.
o
o Coefficient of stability: when the interval between testing
o Variance: standard deviation squared. It is useful because
is greater than 6 months,
it can be broken into components:
- Consider possibility of carryover effect: occurs when first testing
o True variance: variance from true differences  are
session influences scores from the second session
assumed to be stable
- If something affects all the test takers equally, then the results are
o Error variance: random irrelevant sources
uniformly affected and no net errors occurs
- Standard error of measurement: we assume that the distribution of
- Practice tests may make this effect happen
random errors will be the same for all people, classical test theory
- Practice can also affect tests of manual dexterity
uses the standard deviation of errors as the basic measure of error
- Time interval between testing sessions must be selected and
o Standard error of measurement tells us, on the
evaluated carefully
average, how much a score varies from the true score
- Poor test-retest correlations do not always mean that a attest is
o Standard deviation of the observed score and the
unreliable – suggest that the characteristic under study has changed
reliability of the test are used to estimate the standard
error of measurement
PARALLEL-FORM OR ALTERNATE FORMS RELIABILITY
- Reliability: proportion of the total variance attributed to true
- compares two equivalent forms of a test that measure the same
variance.
attribute
o the greater portion of total variance attributed to true
- Two forms should be equally constructed, both format, etc.
variance, the more reliable the test
- When two forms of the test are available, one can compare
- Measurement error: refers to collectively, all of the factors
performance on one form versus the other – equivalent forms
associated with the process of measuring some variable, other than
reliability or parallel forms
the variable being measured - Coefficient of equivalence: degree of relationship between various
o Random error: a source of error in measuring a targeted forms of a test can be evaluated by means of an alternate-forms
variable caused by unpredictable fluctuations and - Parallel forms: each form of the test, the means and variances of
inconsistencies of other variables in the measurement observed test scores are equal
process - Alternate forms: different versions of a test that have been
 This source of error fluctuates from one testing constructed so as to be parallel
situation to another with no discernible pattern - (1) two test administrations with the same group are required
that would systematically raise or lower scores - (2) test scores may be affected by factors such as motivation etc.
o Systematic Error: - Problem: developing a new version of a test
 A source of error in measuring a variable that is INTERNAL CONSISTENCY
typically constant or proportionate to what is - How well does each item measure the content/construct
presumed to be true value of the variable under consideration
being measured - How consistent the items together
 Error is predictable and fixable - Used when tests are administered once
 Does not affect score consistency - If all items on a test measure the same construct, then it has a good
internal consistency
SOURCES OF ERROR VARIANCE - Split-half reliability, KR20, Cronbach Alpha
 CHAPTER 5:
o Test takers with the same score on a heterogeneous test
SPLIT-HALF RELIABILITY may have quite different abilities
- Correlating two pairs of scores obtained from equivalent halves of a o However, homogenous testing is often an insufficient tool
single test administered once. for measuring multifaceted psychological variable such as
- This is useful when it is impractical to assess reliability with two tests intelligence or personality
or to administer test twice
- Results of one half of the test are then compared with the results of Measures of Inter-Scorer Reliability
the other - In some types of tests under some conditions, the score may be more a
- Rules in splitting forms into half: function of the scorer than of anything else
o Do not divide test in the middle because it would lower - Inter-scorer reliability: is the degree of agreement or consistency between
the reliability two or more scorers (or judges or rather) with regard to a particular
o Different amounts of anxiety and differences in item measure
difficulty shall also be considered - Coefficient of inter-scorer reliability: coefficient of correlation to
o Randomly assign items to one or the other half of the test determine the degree of consistency among scorers in the scoring of a test
o use the odd-even system: where one subscore is obtained - Kappa statistic is the best method for assessing the level of agreement
for the odd-numbered items in the test and another for among several observers
the even-numbered items o Indicates the actual agreement as a proportion of the potential
- To correct for half-length, apply the Spearman-Brown formula, which agreement following the correction for chance agreement
allows you to estimate what the correlation between the two halves o Cohen’s Kappa – 2 raters
would have been if each half had been the length of the whole test o Fleiss’ Kappa – 3 or more raters
o Use this if test user wish to shorten a test
HOMOGENEITY VS. HETEROGENEITY OF TEST ITEMS
o Used to determine the number of items needed to attain a
- Homogeneous items has high degree of reliability
desired level of reliability
- Reliability increases as the test length increases
DYNAMIC VS. STATIC CHARACTERISTICS
- Dynamic: trait, state, ability presumed to be ever-changing as a function of
KUDER-RICHARDSON FORMULAS OR KR20/KR21 situational and cognitive experiences
- Kuder-Richardson technique simultaneously considers all - Static: trait, state, ability relatively unchanging
possible ways of splitting the items
- The formula for calculating the reliability of a test in which the items RESTRICTION OR INFLATION OF RANGE
are dichotomous, scored 0 or 1, is the Kuder-Richardson 20 (see - If it is restricted, reliability tends to be lower.
p.114) - If it is inflated, reliability tends to be higher.
- Introduced KR21 – uses an approximation of the sum of the pq
products – the mean test score
SPEED TESTS VS. POWER TESTS
- Speed test: test is homogenous, means that it is easy but short time
CRONBACH ALPHA
- Power test: Few items, but more complex.
- Cronbach developed a formula that estimates the internal
consistency of tests in which the items are not scored as 0 or 1 – a CRITERION-REFERENCED TESTS
more general reliability estimate, which he called coefficient alpha
- Provide an indication of where a testtaker stands with respect to some
- Sum the individual item variances
variable or criterion.
o Most general method of finding estimates of reliability - Tends to contain material that has been mastered in hierarchical fashion.
through internal consistency
- Scores here tend to be interpreted in pass-fail terms.
- Domain sampling: define a domain that represents a single trait or - Measure of reliability depends on the variability of the test scores: how
characteristic, and each item is an individual sample of this general
different the scores are from one another.
characteristic
- Factor analysis deals with the situation in which a test The Domain Sampling Model
apparently measures several different characteristics
- This model considers the problems created by using a limited number
o Good for the process of test construction of items to represent a larger and more complicated construct
- Most widely used as a measure of reliability because it requires - Our task in reliability analysis is to estimate how much error we would
only one administration of the test make by using the score from the shorter test as an estimate of your
- Ranges from 0 to 1 “bigger is always better” true ability
Other Methods of Estimating Internal Consistencies - Conceptualizes reliability as the ratio of the variance of the observed
- Inter-item consistency: refers to the degree of correlation among all score on the shorter test and the variance of the long-run true score
the items on a scale
- Reliability can be estimated from the correlation of the observed test
o A measure of inter-item consistency is calculated from a score with the true score
single administration of a single form of a test
o An index of inter-item consistency, in turn, is useful
Item Response Theory
in assessing the homogeneity of the test
- Classical test theory requires that exactly the same test items be
o Tests are said to be homogenous if they contain items that administered to each person – BAD
measure a single trait
- Item response theory (IRT) is newer – computer is used to focus on
o Definition: the degree to which a test measures a single
the range of item difficulty that helps assess an individual’s ability
factor
level
o Heterogeneity: degree to which a test measures different
o More reliable estimate of ability is obtained using a
factors
shorter test with fewer items
o Ex: homo=test that assesses knowledge only of #-D o Takes a lot of items and effort
television repair skills vs. a general electronics repair test
(hetero)
Generalizability theory
o The more homogenous a test is, the more inter-item
- based on the idea that a persons test scores vary from testing to testing
consistency it can be expected to have
because of variables in the testing situation
o Test homogeneity is desirable because it allows relatively
- Instead of conceiving of all variability in a persons scores as error, Cronbach
straightforward test-score interpretation
encouraged test developers and researchers to describe the details of the
o Test takers with the same score on a homogenous test
particular test situation or universe leading to a specific test score
probably have similar abilities in the area tested
 CHAPTER 5:
- This universe is described in terms of its facets: which include things like
the number of items in the test, the amount of training the test scorers
have had, and the purpose of the test administration
- According to generalizability theory, given the exact same conditions of all
the facets in the universe, the exact same test score should be obtained
- Universe score: the test score obtained and its analogous to a true score in
the true score model
- Cronbach suggested that tests be developed with the aid of
a generalizability study followed by a decision study
- Generalizability study: examines how generalizable scores from a
particular test are if the test is administered in different situations
- How much of an impact different facets of the universe have on the test
score
- Ex: is the test score affected by group as opposed to individual
administration
- Coefficients of generalizability: the influence of particular facts on the test
score is represented by this. These coefficients are similar to reliability
coefficients in the true score model
- Decision study: developers examine the usefulness of test scores in helping
the test user make decision
- The decision study is designed to tell the test user how test scores
should be used and how dependable those scores are as a basis for
decisions, depending on the context of their use

What to Do About Low Reliability

- Two common approaches are to increase the length of the test and to
throw out items that run down the reliability
- Another procedure is to estimate what the true correlation would
have been if the test did not have measurement error
Increase the Number of Items
- The larger the sample, the more likely that the test will represent the
true characteristic
o This could entail a long and costly process however
- Prophecy formula
Factor and Item Analysis
- Reliability of a test depends on the extent to which all of the items
measure one common characteristic
- Factor analysis
o Tests are most reliable if they are unidimensional: one
factor should account for considerably more of the
variance than any other factor
- Or examine the correlation between each item and the total score for
the test
o Called discriminability analysis: when the correlation
between the performance on a single item and the total
test score is low, the item is probably measuring
something different from the other items on the test

Correction for Attenuation

- Potential correlations are attenuated, or diminished, by measurement
error
 CHAPTER 6:
The Concept of Validity (N/2)
- Validity: as applied to a test, is a judgment or estimate of how well a test o CVR Content validity ratio
measures what it purports to measure in a particular context o ne  Number of
o Judgment based on evidence about the appropriateness of panelists stating
inferences drawn from test scores “essential”
o Validity of test must be shown from time to time to account for o N Total number of panelists
culture and advancement  CVR is calculated for each item
- Inference: a logical result or deduction o Culture and the relativity of content validity
- “Acceptable” or “weak” validity of tests and test scores  Tests thought of as either valid or invalid
- Validation: process of gathering and evaluating evidence about validity  What constitutes historical fact depends to some
o Test user and testtaker both have roles in validation of test extent on who is writing the history
o Test users may conduct their own validation studies: may yield  Culture relativity
insights regarding a particular population of testtakers as  Politics (politically correct)
compared to the norming sample (in manual) Criterion-Related Validity
o Local validation studies: absolutely necessary when test user - Criterion-related validity: judgment of how adequately a test score can be
plans to alter in some way the format, instructions, language, or used to infer an individual’s most probable standing on some measure of
content of the test interest (measure of interest being the criterion)
- Types of Validity (Trinitarian view) *not mutually exclusive all - 2 types:
contribute to a unified picture of a test’s validity/ critique approach is o Concurrent validity: index of the degree to which a test score is
fragmented and incomplete related to some criterion measure obtained at the same time
o Content validity: measure of validity based on an evaluation of (concurrently)
the subjects, topics, or content covered by the items in the test o Predictive validity: index of the degree to which a test score
o Criterion-related validity: measure of validity obtained by predicts some criterion measure
evaluating the relationship of scores obtained on the test to - What Is a Criterion?
scores on other tests or measures o Criterion: a standard on which a judgment or decision may be
o Construct validity: measure of validity that is arrived at by based; standard against which a test or a test score is
executing a comprehensive analysis of: (umbrella validity evaluated (criterion-related validity)
every other variety of validity falls under it) o Characteristics of criterion
 How scores on test relate to other test scores  Relevancy pertinent or applicable to the matter
and measures at hand
 How scores on test can be understood within some  Validity (for the purpose which it is being used)
theoretical framework for understand the  Uncontaminated Criterion contamination: term
construct that the test was designed to measure applied to a criterion measure that has been based,
- Strategies: ways of approaching the process of test validity at least in part, on predictor measures
o Content validation strategies - Concurrent Validity
o Criterion-related validation strategies o Test scores are obtained at about the same time as the criterion
o Construct validation strategies measures are obtained measures of the relationship
- Face Validity between the test scores and the criterion provide evidence of
o Face validity: relates more to what a test appears to measure to concurrent validity
the person being tested than to what the test actually measures o Indicate the extent to which test scores may be used to
o Judgment concerning how relevant the test items appear estimate an individuals present standing on a criterion
to be usually from testtaker, not test user o Once validity of inference from test scores is established=
o Lack of face validity= lack of confidence in faster, less expensive way to offer a diagnosis or a classification
perceived effectiveness of test which decreases decision
testtaker’s motivation/cooperation *may still be o Concurrent validity of a test can be explored with respect
useful to another test
- Content validity  Prior research must have satisfactorily demonstrated
o Content validity: a judgment of how adequately a test samples the 1st test’s validity
behavior representative of the universe of behavior that the test  1st test= validating criterion
was designed to sample - Predictive validity
 Ideally, test developers have a clear vision of the o Test scores may be obtained at one time and the criterion
construct being measured clarity reflected in measures obtained at a future time, usually after some
the content validity of the test intervening event has taken place
o Test blueprint: structure of the evaluation; a plan regarding the  Intervening event training, experience,
types of information to be covered by the items, the number of therapy, medication, etc.
items tapping each area of coverage, the organization of the  Measures of relationship between the test scores
items in the test, etc. and a criterion measure obtained at a future time
 Behavior observation is a technique frequently used provide an indication of the predictive validity test
in test blueprinting (how accurately scores on the test predict some
o The quantification of content validity criterion measure)
 Important in employment settings  tests used to o Ex: SAT test score and freshman gpa
hire and promote o Judgments of criterion validity are based on 2 types of statistical
 One method: method for gauging agreement among evidence:
raters or judges regarding how essential a particular  The validity coefficient
item is (C.H. Lawshe)  Validity coefficient: correlation
 “Is the skill or knowledge measured by coefficient that provides a measure of the
this item… relationship between test scores and
o Essential scores on the criterion measure
o Useful but not essential  Ex: Pearson correlation coefficient used
o Not necessary to determine validity between 2 measures
 To the performance of the job?” (r)
 Content validity ratio (CVR):  Affected by restriction or inflation of
 CVR= ne – (N/2) range
 CHAPTER 6:
 Is the range of scores employed  Construct: an informed, scientific idea developed or
appropriate to the objective of the hypothesized to describe or explain behavior
correlational analysis  Ex: intelligence, depression, motivation,
 No rules regarding the validity coefficient personality, etc.
(how high or low it should/could be for  Unobservable, presupposed (underlying)
test to be valid) traits that a test developer invokes to
 Incremental validity describe test behavior/criterion
o More than one predictor performance
o Incremental validity: the  Viewed as unifying concept for all validity evidence
degree to which an additional o Evidence of Construct Validity
predictor explains something  Various techniques of construct validation that
about the criterion measure provide evidence:
that is not explained by  Test is homogeneous measures single
predictors already in use construct
 Expectancy data  Test scores increase/decrease as function
 Expectancy data: provides info that can of age, passage of time, or experimental
be used in evaluating the criterion-related manipulation (theoretically predicted)
validity of a test  Test scored obtained after some even or
 Score obtained on expectancy passage of time differ from pretest scores
test/tables likelihood testtaker will (theoretically predicted)
score within some interval of scores on a  Test scores obtained by people from
criterion measure (“passing”, distinct groups vary (theoretically
“acceptable”, etc.) predicted)
 Expectancy table: shows the percentage  Test scores correlate with scores on other
of people within specified test-score tests (theoretically predicted)
intervals who subsequently were placed  Evidence of homogeneity
in various categories of the criterion  Homogeneity: refers to how uniform a
o May be created from test is in measuring a single concept
scatterplot  Evidence correlations between subtest
o Shows relationships scores and total test scores
 Expectancy chart: graphic representation  Item-analysis procedures have been used
of an expectancy table in quest for test homogeneity
o The higher the initial rating,  Desirable but not necessary
the greater the probability  Contributes no info about how construct
of job/academic success being measured relates to other
 Taylor Russell Table – provide an estimate of the constructs
extent to which inclusion pf a particular test in the  Evidence of changes with age
selection system will actually improve selection  If test purports to measure a construct
 Selection ratio – relationship between the that changes over time then the test
number of people to be hired and the scores, too, should show progressive
number of people available to be hired changes to be considered valid
 Base rate – percentage of people under measurement of construct
existing system for a particular position  Does not in itself provide info about how
 Relationship between predictor and construct relates to other constructs
criterion must be linear  Evidence of pretest-posttest changes
 Naylor-shine Tables – difference between the means  Can be evidence of construct validity
of the selected and unselected groups to derive an  Some more typical intervening
index of what the test is adding to already experiences responsible for changes in
established procedures test scores are:
o Decision theory and Test utility o Formal education
 Base rate – extent to which a particular trait, o Therapy/medication
behavior, characteristic or attribute exists in the o Any life experience
population  Evidence from distinct groups/method of contrasted
 Hit rate – defined as the proportion of people a test groups
accurately identifies as possessing or exhibiting a  Method of contrasted groups: one way of
particular trait. providing evidence for the validity of a
 Miss rate – proportion of people the test fails to test is to demonstrate that scores on the
identify as having or not having attributes test vary in a predictable way as a
 False positive (type I error) – possess function of membership in some group
particular attribute but actually does not  Rationale if a test is a valid measure of
have. Ex: score above cutoff score, hired a particular construct, test scores from
but failed the job. groups of people who would presumed
 False negative (type II error) – does not with respect to that construct should have
possess particular attribute but actually correspondingly different test scores
does have. Ex. Scored below cutoff score,  Convergent evidence
not hired, but could have been successful  Evidence for the construct validity of a
in the job particular test may converge from a
- Construct Validity number of sources, such as tests or
o Construct validity: judgment about the appropriateness of measures designed to assess the
inferences drawn from test scores regarding individual standings same/similar construct
on a variable called a construct
 CHAPTER 6:
 Convergent evidence: scores on a test  Criterion data may be influenced by
undergo construct validity and correlate rater’s knowledge of ratee race,
highly in the predicted direction with gender, etc.
scores on older, more established and o Test fairness
already validated tests designed to  Issues of fairness tend to be more difficult and
measure the same/similar construct involve values
 Discriminant evidence  Fairness: the extent to which a test is used in an
 Discriminant evidence: validity coefficient impartial, just, and equitable way
showing little relationship between test  Sources of misunderstanding
scores and /or other variables with which  Discrimination
scores on the test being construct-  Group not included in standardization
validated should not theoretically be sample
correlated  Performance differences between
 Provides evidence of construct validity identified groups
 Multitrait-multimethod matrix: “two or
more traits”, “two or more methods” Relationship Between Reliability and Validity
matrix/table that results from correlating - A test should not correlate more highly with any other variable than
variables (traits) within and between it correlates with itself
methods - A modest correlation between the true scores on two traits may be
 Factor analysis missed if the test for each of the traits is not highly reliable
 Factor analysis: shorthand term for a class - We can have reliability without validity
of mathematical procedures designed to o It is impossible to demonstrate that an unreliable test
identify factors or specific variables that is valid
are typically attributes, characteristics, or
dimension on which people may differ
 Frequently used as a data reduction
method in which several sets of scores
and correlations between them are
analyzed
 Exploratory factor analysis: researchers
test the degree to which a hypothetical
model fits the actual data
o Factor loading: conveys
information about the extent
to which the factor determines
the test score or scores
o Complex procedures
- Validity, Bias, and Fairness
o Test Bias
 Bias: a factor inherent in a test that systematically
prevents accurate, impartial measurement
 Technical means to identify and remedy bias
(mathematically)
 Bias implies systematic variation
 Rating error
 Rating: a numerical or verbal judgment
(or both) that places a person or an
attribute along a continuum identified by
a scale of numerical or word descriptions,
known as a rating scale
 Rating error: judgment resulting from
intentional or unintentional misuse of a
rating scale
 Leniency error/generosity error: error in
rating that arises from the tendency on
the part of the rater to be lenient in
scoring, marking, and/or grading
 Severity error: rater exhibits general and
systematic reluctance to giving ratings at
either the positive or negative extreme
 Overcome restriction of range rating errors is to use
rankings: procedure that requires the rater to
measure individuals against one another instead of
against an absolute scale
 Rater is forced to select 1st, 2nd, 3rd, etc.
 Halo effect: fact that for some raters, some rates can
do no wrong
 Tendency to give a particular ratee a
higher rating than he or she objectively
deserves
 CHAPTER 7: UTILITY
Utility: usefulness or practical value of testing to improve efficiency  Based on norm-related considerations rather
than on the relationship of test scores to a
Factors that Affect a Test’s Utility criterion
 Psychometric Soundness  Also called norm-referenced cut score
o Reliability and validity of a test  Ex.) top 10% of test scores get A’s
o Gives us the practical value of both the scores (reliability o Fixed cut score: set with reference to a judgment
and validity) concerning a minimum level of proficiency required to be
o They tell us whether decisions are cost-effective included in a particular classification.
o A valid test is not always a useful test  Also called absolute cut scores
 especially if testtakers do not follow test o Multiple cut scores: using two or more cut scores with
directions reference to one predictor for the purpose of categorizing
 Costs testtakers
o Economic and non economic  Ex.) having cut score that marks an A, B, C etc.
o Ex.) using a less expensive and therefore less stringent all measuring same predictor
application process for airline personnel. o Multiple hurdles: for success, requires one individual to
 Benefits complete many tasks, with elimination at each level
o Profits, gains, advantages  Ex.) written application group interview
o Ex.) more stringent hiring policy more personal interview etc.
productive employees o Compensatory model of selection: assumption is made
o Ex.) maintaining successful and academic environment that high scores on one attribute can compensate for low
of university scores on another attribute

Utility Analysis Methods for Setting Cut Scores

What is Utility Analysis? The Angoff Method

-a family of techniques that entail a cost-benefit analysis designed to yield Judgments of experts are averaged
information relevant to a division about the usefulness and/or practical value of
a tool of assessment. The Known Groups Method
Collection of data on the predictor of interest from group known to posses and
Utility analysis: An illustration not to possess trait, attribute, or ability
What’s the companies goal? Cut score based on which test best discriminates the two groups performance
 Limit the cost of selection
o Don’t use FERT IRT-Based Method
 Ensure that qualified candidates are not rejected Based on testtaker’s performance across all items on a test
o Set a cut score that yields the lowest false negative rate Some portion of test items must be correct
 Ensure that all candidates selected will prove to be qualified Item-mapping method: determining difficulty level reflected by cut score (?)
o Lowest dales positive rate Book-Mark method: test items are listed, one per page, in ascending level of
 Ensure, to the extent possible, that qualified candidates will be difficulty. An expert places a bookmark to mark the divide which separates
selected and unqualified candidates will be rejected testtakers who have acquired minimal knowledge, skills, or abilities and those
o False positives are no better or worse than false negatives that have not.
o Highest hit rate and lowest miss rate Problems include training of experts, possible floor and ceiling effects, and the
optimal length of item booklets
How Is a Utility Analysis Conducted?
-objective: dictate what sort of information will be required as well as the Other Methods
specific methods to be used -discriminant analysis: family of statistical techniques used to shed light on the
 Expectancy Data relationship between certain variables and two or more naturally occurring
o Expectancy table provides indication of the likelihood that groups
a testtaker will score within some interval of scores on a ex.) the relationships between scores of tests and ppl judged to be
criterion measure successful or unsuccessful at job
o Used to measure costs vs. benefits
 Brogden-Cronbach-Gleser formula
o Utility gain: estimate of the benefit of using a particular
test or selection method
o Most simply is benefits-cost
o Productivity gain: estimated increase in work output

Some Practical Considerations

 The Pool of Job Applicants
o There is rarely a limitless supply of potential employees
o Dependent on many factors, including economic
environment
o We assume that top scoring individuals will accept the job,
but those individuals are more likely to be the ones being
offered higher positions
 The complexity of the Job
o It is questionable whether the same utility analysis
methods can be used to measure the eligibility of varying
complexities of jobs
 The cut score in use
o Relative cut score: may be defines as reference point
 CHAPTER 8: TEST
CHAPTER 8: TEST DEVELOPMENT o Item format: variables such as the form, plan, structure,
arrangement and layout of individual test items
STEPS: o 2 types
1.) selected-response format: testtaker selects a response from
1. TEST CONCEPTUALIZATION o
a set of alternative responses
2. TEST CONSTRUCTION
3. TEST TRYOUT  includes multiple choice, true-false, and matching
2.) constructed-response format: testtaker supplies or creates
4. ITEM ANALYSIS o
5. TEST REVISION the correct answer
 includes completion item, short answer and essay
TEST CONCEPTUALIZATION - Writing Items for computer administration
- Thoughts or stimulus that could be almost everything. o Item bank: relatively large and easily accessible collection of
- An emerging social phenomenon or pattern of behavior might serve test questions
as the stimulus for the development of a new test. o Computerized Adaptive Testing (CAT): interactive, computer-
- Norm referenced: An item for which high scorers on the test administered testtaking process wherein items presented to the
respond correctly. Low scorers respond to that same item incorrectly testtaker are based in part on testtaker’s performance on
- Criterion referenced: high scorers on the test get a particular item previous items.
right whereas low scorers on the test get that same item wrong. o Floor effect: the diminished utility of an assessment tool for
- Pilot work: pilot study or pilot research. To know whether some items distinguishing testtakers at the low end of the ability, trait, or
should be included in the final form of the instrument. other attribute being measured
o the test developer typically attempts to determine how o Ceiling effect: diminished utility of an assessment tool for
best to measure a targeted construct distinguishing testtakers at the high end of the ability, trait,
TEST CONSTRUCTION attribute being measured
- Scaling: process of setting rules for assigning numbers in o Item branching: ability of computer to tailor the content and
measurement. order of presentation of test items on the basis of responses to
- L.L. Thurstone: credited for being the forefront of efforts to develop previous items
methodologically sound scaling methods. SCORING ITEMS
TYPES OF SCALES: - Cummulative scoring: testtakers earn cumulative credit with regard to
- Nominal, ordinal, interval or ratio a particular construct
- Age-based scale - Class/category scoring: testtaker responses earn credit toward
- Grade-based scale placement in a particular class or category with other testtakers
- Stanine scale (raw score converted to 1-9) whose pattern of responses is presumably similar in some way
- Unidimensional vs. multidimensional - Ipsative scoring: comparing a testtaker’s score on one within a test to
o Unidimensional: measuring one construct another scale within that same test
o Multidimensional: measuring more than one construct o ex.) “John’s need for achievement is higher than his need
- Comparative vs. categorical for affiliation”
o Comparative scaling: entails judgments of a stimulus in ITEM WRITING (KAPLAN BOOK)
comparison with every other stimulus on the scale Item Writing
o Categorical scaling: stimuli are placed into one of two or - Personality and intelligence tests require different sorts of responses
more alternative categories that differ quantitatively with - Guidelines for item writing
respect to some continuum o Define clearly what you want to measure
- Rating Scale: Which can be defined as a grouping of words, o Generate an item pool
statements, or symbols on which judgments of the strength of a o Avoid exceptionally long items
particular trait, attitude, or emotion are indicated by the testtaker o Keep the level of reading difficulty appropriate for those who
- Summative scale: when final score is obtained by summing the will complete the scale
ratings across all the items o Avoid “double-barreled” items that convey two or more ideas
- Likert scale: each item presents the testtaker with five alternative at the same time
responses usually on agree-disagree, or approve-disapprove o Consider mixing positively and negatively worded items
continuum - Must be sensitive to ethnic and cultural differences
- Method of paired comparisons: presented with two stimuli and - Items that retain their reliability are more likely to focus on skills, while
asked to compare those that lost reliability focused on more abstract concepts
- Comparative scaling: judging of a stimulus in comparison with every Item Formats
other stimulus on the scale - Simplest test uses dichotomous format
- Categorical scaling: testtaker places stimuli into a category; those The Dichotomous Format
categories differ quantitatively on a spectrum. - Dichotomous format offers two alternatives for each item
- Guttman scale (Scalogram analysis): items range from sequentially o Ie. True-false examination
weaker to stronger expressions of attitude, belief, or feeling. A - Advantages:
testtaker who agrees with the stronger statement is assumed to also o Simplicity
agree with the milder statements o True-false items require absolute judgment
- Equal-appearing intervals (Thurstone): direct estimation because - Disadvantages:
don’t need to transform testtaker’s response to another scale o True-false encourage students to memorize material
WRITING ITEMS o “truth” often comes in shades of gray
- 3 Questions of test developer o mere chance of getting any item correct is 50%
o What range of content should the items cover? - Yes-no format on personality tests
o Which of the many different types of item formats should be - Multiple-choice = polytomous
employed? The Polytomous Format
o How many items should be written in total and for each content - Polytomous format resembles the dichotomous format except that each
area covered? item has more than two alternatives
- Item pool: reservoir from which items will not be drawn for the final o Multiple-choice exams
version of the test (should be about double the number of questions as - Advantage:
final will have) o Little time for test takers to respond to a particular item
- Item format because they do not have to write
- Incorrect choices are called distractors
 CHAPTER 8: TEST
- Disadvantages: o The midpoint representing the optimal difficulty is
o How many distractors should a test have? --> 3 or 4 obtained by summing up the chance of success proportion
o Distractors hurting reliability / validity of test and 1.00 and then dividing the sum by 2
o Three alternative multiple-choice items may be better than five Item Reliability Index
alternative items because they retain the psychometric value o Indication of the internal consistency of a test
but take less time to develop and administer o Equal to the product of the item-score standard deviation (s) and the
o Scoring of the MC exams? --> simply guessing should elicit correlation (r)
correctness o Factor analysis and inter-item consistency
o Correcting for this though, the expected score is 0 – as getting a o Factor analysis determines whether items on a test appear
question wrong loses you a point to be measuring the same thing
- Guessing can be good if you can narrow down a couple answers The Item-Validity Index
- Students are more likely to guess when they anticipate a lower grade on a o Statistic designed to provide an indication of the degree to which a
test than when they are more confident test is measuring what it purports to measure
- Guessing threshold describes the chances that a low-ability test taker will o Requires: item-score standard deviation, the correlation between the
obtain each score item score and criterion score
- True-false and MC tests are common to educational and achievement tests The Item-Discrimination Index
- Likert format, category scale, and the Q-sort used for personality-attitude o Measures how adequately an item separates or discriminates
tests between high scorers and low scorers
Likert Format o “d”
- Likert format: requires that a respondent indicate the degree of agreement o compares performance on a particular item with performance in the
with a particular attitudinal question upper and lower regions of a distribution of continuous test scores
o Strongly disagree ... Strongly agree o higher d means greater number of high scorers answering the item
o For measurements of attitude correctly
- Used to create Likert Scales: scales require assessment of item o negative d means low-scoring examinees are more likely to answer
discriminability the item correctly than high-scoring examinees
- Familiar and easy --- likely to remain popular in personality and attitude o Analysis of item alternatives
tests Item-Characteristic Curves?
Category Format o Graphic representation of item difficulty and discrimination
- Category format: uses more choices than Likert; 10-point rating scale
- Disadvantage: responses to items on 10-pt scales are affected by the Other Considerations in Item Analysis
groupings of the people or things being rated o Guessing
- People change their ratings depending on context o Usually in some direction
o This problem can be avoided if the endpoints of the scale are o Depends on individuals ability to take risks
clearly defined and the subjects are frequently reminded of the o Item fairness
definitions of the endpoints o Bias
- Optimal number of points is 7? o Speed tests
o Number depends on the fineness of the discrimination that o Last items will appear to be more difficult because
subjects are willing to make not everyone got to them
o When people are highly involved with some issue, they will tend
to respond best to a greater number of categories Qualitative Item Analysis
- Increasing the number of response categories may not increase reliability  Qualitative methods: techniques of data generation and analysis that
and validity rely primarily on verbal rather than mathematical or statistical
- Visual analogue scale: respondent is given a 100-millimeter line and asked procedures
to place a mark between two well-defined endpoints  Qualitative item analysis: various nonstatistical procedures designed
o Measures self-rate health to explore how individual test items work
Checklists and Q-Sorts o Through means like interviews and group discussions
- Adjective Checklist: subject receives a long list of adjectives and indicates  “Think aloud” test administration
whether each one is characteristic of himself or herself o approach to cognitive assessment that entails respondents
o Requires subjects either to endorse such adjectives or not, thus vocalizing thoughts as they occur
allowing only two choices for each item o used to shed light on the testtker’s though processes
- Q-Sort: increases the number of categories during the administration of a test
o Used to describe oneself or to provide ratings of others  Expert panels
Other Possibilities o Sensitivity review: study of test items in which they are
- Forced-choice and Likert formats are clearly the most popular examined for fairness to all prospective testtakers as well
in contemporary tests and measures as for the presence of offensive language, stereotypes, or
- Checklists have fallen out of favor because they are more prone to situations
error than are formats that require responses to every item ITEM ANALYSIS (KAPLAN BASED)
- Frequent advice is to not use “all of the above” as a response option The Extreme Group Method
- Compares people who have done well with those who have done
TEST TRYOUT poorly on a test
What is a good item? - Difference between these proportions is called the discrimination
o Reliable and valid index
o Helps to discriminate testtakers The Point Biserial Method
- Find the correlation between performance on the item and
ITEM ANALYSIS performance on the total test
o The Item-Difficulty Index - Correlation between a dichotomous variable and a continuous
o Obtained by calculating the proportion of the total number variable is called a point biserial correlation
of testtakers who answered the item correctly “p” - On tests with only a few items, using this is problematic because
o Higher p= easier item performance on the item contributes to the total test score
o Difficulty can be replaced with endorsement in non- Pictures of Item Characteristics
achievement tests - Valuable way to learn about items is to graph their characteristics,
which you can do with the item characteristic curve
 CHAPTER 8: TEST
- Prepare a graph for each individual test item - First step in developing these tests involves clearly specifying the
o Total test score is used as an estimate of the amount of a objectives by writing clear and precise statements about what the
‘trait’ possessed by individuals learning program is attempting to achieve
- Relationship between performance on the item and performance on - To evaluate the items: one should give the test to two groups of
the test gives some info about how well the item is tapping the info students – one that has been exposed to the learning unit and one
we want that has not
Drawing the Item Characteristic Curve - Bottom of the V is the antimode – the least frequent score
- To draw this, we need to define discrete categories of test - This point divides those who have been exposed to the unit from
performance those who have not been exposed and is usually taken as the cutting
- If the test has been given to many people, we might choose to make score or point, or what marks the point of decision
each test score a single category - When people get scores higher than the antimode, we assume that
- Gradual positive slope of the line demonstrates that the proportion of they have met the objective of the test
people who pass the item gradually increases as test scores increase Limitations of Item Analysis
o This means that the item successfully discriminates at all - Main Problem: though statistical methods for item analysis tell the
levels of test performance test constructor which items do a good job of separating students,
- Ranges in which the curve changes suggest that the item is sensitive, they do not help the students learn
while flat ranges suggest areas of low sensitivity - Although the data are available to give the child feedback on the
- Item analysis breaks the general rule the increasing the number of “bug” in their thinking, nothing in the testing procedure initiates this
items makes a test more reliable guidance
- When bad items are eliminated, the effects of chance responding can TEST REVISION
be eliminated and the test can become more efficient, reliable, and Test Revision in the Life Cycle of an Existing Test
valid  Tests get old and need revision
Item Response Theory  Questions arise over equivalence of two tests
- According to classical test theory, a score is derived from the sum of  Cross-validation and Co-validation
an individual’s responses to various items, which are sampled from a o Cross-validation: revalidation of a test on a sample of
larger domain that represents a specific trait or ability testtakers other than those on whom test performance
- New approaches consider the chances of getting particular items right was originally found to be a valid predictor of some
or wrong – item response theory – make extensive use of item criterion
analysis o Validity shrinkage: decrease in item validities that
o With this, each item on a test has its own item inevitably occurs after cross-validation of finding
characteristic curve that describes the probability of o Co-validation: test validation process conducted on two or
getting each particular item right or wrong given the ability more tests using the same sample of testtakers
level of each test taker o Co-norming: when co-validation is used in conjunction
o Testers can make an ability judgment without subjecting with the creation of norms or the revision of existing
the test taker to all of the test items norms
- Technical adv: builds on traditional models of item analysis and can o Quality assurance during test revision
provide info on item functioning, the value of specific items, and the  test givers must have some degree of
reliability of a scale qualification, training, and testing
- Two dimensions used are difficulty and discriminability  anchor protocol: test protocol scored by a
- Most attractive adv. Is that one can easily adapt the IRT tests for highly authoritative scorer that is designed as a
computer administration model for scoring and a mechanism for
o Computer can rapidly identify the specific items that are resolving scoring discrepancies
required to assess a particular ability level  scoring drift: a discrepancy between scoring in
- “peaked conventional” an anchor protocol and the scoring of another
- “rectangular conventional” – requires that test items be selected to protocol
create a wide range in level of difficulty The Use of IRT in Building and Revising Tests
o problem: only a few items of the test are appropriate for (item response theory)
individuals at each ability level; many test takers spend  Evaluating the properties of existing tests and guiding test revision
much of their time responding to items either  Determining measurement equivalence across testtaker populations
considerably below their ability level or too difficult to o Differential item functioning (DIF): phenomenon, wherein
solve an item functions differently in one group of testtakers as
- IRT addresses traditional problems in test construction well compared to another group of testtakers known to have
- IRT can identify respondents with unusual response patterns and the same level of the underlying trait
offer insights into cognitive processes of the test taker  Developing item banks
- May also reduce the biases against the people whoa re slow o Items from other instruments item pool  scrutiny
in completing test problems preliminary item bank psychometric testingitem bank
External Criteria
- Item analysis has been persistently plagued by researchers’ continued
dependence on internal criteria, or total test score, for evaluating
items
Linking Uncommon Measures
- One challenge in test applications is how to determine linkages
between two different measures
Items for Criterion-Referenced Tests
- Traditional use of tests requires that we determine how well
someone has done on a test by comparing the person’s performance
to that of others
- Criterion-referenced tests compares performance with some clearly
defined criterion for learning
o Popular approach in individualized instruction programs
o Regarded as diagnostic instruments
 CHAPTER 9: INTELLIGENCE AND ITS
What is Intelligence?  group factors: neither as general as g nor as specific as s
Intelligence: a multifaceted capacity that manifests itself in different ways across o ex.) linguistic, mechanical, arithmetical abilities
the lifespan. Usually includes abilities to:  Guilford: multiple-factor models of intelligence
 Acquire and apply knowledge o Explain mental activities by deemphasizing, any reference
 Reason logically to g
 Plan effectively  Thurstone: conceived intelligence as being composed of 7 primary
 Infer perceptively abilities.
 Make judgment and solve problems  Gardner: developed theory of multiple intelligences
 Grasp and visualize concepts o Question over whether emotional intelligence exists.
 Pay attention o Logical-mathematical, bodily-kinesthetic, linguistic,
 Be intuitive musical, spatial, interpersonal and intrapersonal
 Find the right words and thoughts with facility  Raymond Cattell: fluid vs. crystallized intelligence
 Cope with, adjust to, and make the most of new situations o Crystallized intelligence: acquired skills and knowledge
Intelligence Defines: Views of the Lay Public and their retrieval. Retrieval of information and
 Both social and academic application of general knowledge
Intelligence Defined: Views of Scholars and Test Professionals o Fluid intelligence: nonverbal, relatively culture-free, and
 Francis Galton independent of specific instruction.
o First to publish on heritability of intelligence  Horn: added more to 7 factors
o Most intelligent persons were those with the best o Vulnerable abilities: decline with age and tend to return
sensory abilities preinjury levels following brain damage
 Alfred Binet o Maintained abilities: tend not to decline with age and may
o Made tests about intelligence, but didn’t define it return to preinjury levels following brain damage.
o Components of intelligence: reasoning, judgment,  Carrol:
memory, abstraction o Three-stratum theory of cognitive abilities: like geology
o Added that definition is complex; requires interaction of o Hierarchical model: meaning that all of the abilities listed
components in a stratum are subsumed by or incorporated in the
o He argued that when one solves a particular problem, the strata above.
abilities used cannot be separated because they interact o Those in the first stratum are narrow abilities
to produce the solution.  CHC model (Cattell-Horn-Carroll)
 David Wechsler o Some overlap some difference
o Best way to measure this global ability was by measuring o Doesn’t use g
aspects of several “qualitatively differentiable” abilities o Has broader abilities than Carroll’s theory
o Complexity of intelligence  McGrew: Integrated the Cattell-Horn and Carroll’s model
o Conceptualization as an “aggregate” or “global” capacity  McGrew and Flanagan: integrated McGrew-Flanagan CHC Model
 Jean Piaget o Features 10 broad stratum abilities
o Studied children o 70 narrow-stratum abilities
o Believed order of maturation to be unchangeable o Makes no provision for the general intellectual ability
o With age, increased schema: organized action or mental factor (g)
structure that, when applied to the world, leads to o It was omitted because it has little practical relevance to
knowing or understanding. cross-battery assessment and interpretation
o Learning occurred through assimilation (actively The Information-Processing View
organizing new information so that it fits in with what  Aleksandr Luria
already is perceived and thought) and accommodation o How (not what) information is processed
(changing what is already perceived or though so that it o Simultaneous/parallel processing: integrated all at once
fits with new information) o Successive/sequential processing: each bit individually
o Sensorimotor (0-2) processed
 PASS model: (Planning, attention, simultaneous, successive)-model of
o Preoperational (2-6) assessing intelligence
 Sternberg ‘The essence of intelligence is that it provides a means to
o Concrete Operational (7-12) govern ourselves so that our thoughts and actions are organized,
coherent, and responsive to both out internally driven needs and to
o Formal Operational (12 and older) the needs of the environment”

 All share interactionism: complex concept by which heredity and Measuring Intelligence
environment are presumed to interact and influence the
development of one’s intelligence Types of Tasks Used in Intelligence Test
 Factor-analytic theories: focus is squarely on identifying the  Infants: test sensorimotor, interviews with parents
ability(ies) deemed to constitute intelligence  Older child: verbal and performance abilities
 Information-processing theories: focus is on identifying the specific  Mental Age: index that refers to chronological age equivalent to
mental processes that constitute intelligence. one’s test performance
 Adults: retention of general information, quantitative reasoning,
Factor-Analytic Theories of Intelligence: expressive language and memory, and social judgment
 Charles Spearman: pioneered new techniques to Theory in Intelligence Test Development and Interpretation
measure intercorrelations between tests.  Weschler made a dichotomous test (Performance and Verbal), but
o Existence of a general intellectual ability factor (g) that advocated multifaceted definition
tapped by all other mental abilities.  Thorndike: intelligence = social, concrete, abstract
 g representing the portion of the variance that all intelligence tests  Putting theories into test are extremely hard
have in common and the remaining portions of the variance being
accounted for either by specific components (s) or by error Intelligence: Some Issues:
components (e) Nature vs. Nurture
 greater g = better test was thought to predict overall intelligence  Currently believed to be mix of two
 CHAPTER 9: INTELLIGENCE AND ITS
 Performationism: all structures, including intelligence are had at birth
and can’t be improved upon
 Led to predeterminism: one’s abilities are predetermined by genetic
inheritance and no learning or intervention can enhance it
 Interactionist: ppl inherit certain intellectual potential
o Theres a limit to genetic abilities (i.e. can’t ever have x-ray
vision)
The Stability of Intelligence
 Stable pretty much throughout one’s adult life
 Cognitive abilities seem to decline with age
The Construct Validity of Tests of Intelligence
 Having construct validity requires having unified understanding of
what intelligence is
 Very difficult. Spearman says its one thing, Guilford says its many
 Thorndike approach is sort of compromise
o Look for one central factor with three additional factors
representing social, concrete, and abstract intelligences
Other Issues
 Flynn effect: IQ scores seem to rise every year, but not coupled with
rise in “true intelligence”
 Personality
o High IQ: Need for achievement, competition, curiosity,
confidence, emotional stability etc.
o Low IQ: passivity, dependence, maladjustment
o Temperament (used to describe infants)
 Gender
o Men usually outscore in visual spatialization tasks and
intelligence scores
o Women tend to outscore in language-skill tasks
o But differences can be bridged
 Family Environment
o Divorce can have negative effects
o Begins with “maternal effects” in womb
 Culture
o Provides specific models for thinking, acting and feeling
o Assumed that if cultural factors can be controlled then
differences between cultural groups will be lessened
o Assumed that culture can be removed by the reliance on
exclusively nonverbal tasks
 Tend not to be very good at predicting success
in various academic and business settings
o Culture loading: the extent to which a test
incorporates the vocabulary, concepts, traditions,
knowledge and feelings associated with a particular
culture
o No test can be culture free
o Culture-fair intelligence test: test/assessment process
designed to minimize the influence of culture with regard
to various aspects of evaluation procedure
o Another approached called for cultural-specific intelligence
tests
 Ex.) BITCH measured streetwiseness
 Lacked predictive validity and useful, practical
information
 CHAPTER 10: TESTS OF
The Stanford-Binet Intelligence Scales Other Measures of Intelligence
 First to have detailed administration and scoring instructions Tests Designed for Individual Administration
 First American test to test IQ  Kaufman Adolescent and Adult Intelligence Test
 First to use alternate items (an item that can be used in place of  Kaufman Brief Intelligence Test
another)  Kaufman Assessment Battery for Children
 Lacked minority group representation  Away from information processing and towards a distinction
 Ratio IQ=(mental age/chronological age)x100 between sequential and simultaneous processing
 Deviation Ratio/test composite: performance of one individual Tests Designed for Group Administration
compared to the performance of others of the same age. Has  Group Testing in the Military
mean of 100 and standard deviation of 16 o WWI need for government to test intelligence
 Age scale: items grouped by age as means of differentiating “unfit” and
 Point scale: items organized by category “exceptionally superior ability”
The Stanford-Binet Intelligence Scales: Fifth Edition o Army Alpha Test: to army recruits who could read.
 Measures fluid intelligence, crystallized knowledge, quantitative Included general information questions, analogies, and
knowledge, visual-processing, and short-term (working) memory scrambled sentences to reassemble
 Utilizes adaptive testing: testing individually tailored to testtakers o Army Beta Test: to foreign or illiterate recruits,
to ensure that items are neither too difficult (frustrating) or too included mazes, coding, and picture completion.
easy (false hope) o After the war, the alpha and beta test were used
 Examiner establishes rapport with testtaker, then administers rampantly, and oftentimes misused
routing test to direct, route examinee to test items most likely at o Screening tools: instrument of procedure used to
optimal level of difficulty identify a particular trait or constellation of traits
 Teaching items: show testtaker what is expected, how to do it. o ASVAB (Armed Services Vocational Aptitude Battery):
o Can be used for qualitative assessment, but not scoring administered to prospective to recruits or high school
 Subtests for verbal and nonverbal tests share same name, but students looked for career guidance
involve different tasks  5 career areas: clerical, electronics,
 Floor: lowest level of items on subtest mechanical, skill-technical, and combat
 Ceiling: highest-level item of subtest operations
 Basal level: base-level criterion that must be met for testing on  Group Testing in Schools
the subtest to continue o Useful in developing child’s profile- but cannot be sole
 Ceiling level is met when testtaker fails certain number of items in indicator
a row. Test discontinues here. o Groups of 10-15
 Scores: raw standard  composite o Starting in Kindergarten
 Extra-test behavior: behavioral observation o Also called traditional group testing, because more
The Wechsler Tests modern forms can utilize computer. These more aptly
-commonality between all versions: all yield deviation IQ’s with mean of 100 called individual testing
and standard deviation of 15 Measures of Specific Intellectual Abilities
Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV)  Widely used intelligence tests only test a sampling of the many
 Core subtest: administered to obtain a composite score attributable factors aiding in intelligence
 Supplemental/Optional Subtest: provides additional clinical  Ex.) Creativity
information or extending the number of abilities or processes o Commonly thought to be composed of originality,
sampled. fluency, flexibility, and elaboration
o If the focus is too heavily on whether an answer is
 Yields four index scores: Verbal Comprehension Index, a Working
correct, doesn’t allow for creativity
Memory Index, a Perceptual Reasoning Index, and a Processing
o Achievement tests require convergent thinking:
Speed Index
The Wechsler Intelligence Scale for Children –Fourth Edition (WISC-IV) deductive reasoning process that entails recall and
consideration of facts as well as a series of logical
 Process score: index designed to help understand how testtakers
judgments to narrow down solutions and eventually
process various kinds of information
arrive at one solution
 WISC-IV compared to the SB5
o Divergent thinking: a reasoning process in which
The Wechsler Preschool and Primary Scale of Intelligence-Third Edition
thought is free in many different directions, making
(WPPSI-III)
several solutions possible
 New school for children under 6
 Associated words, uses of rubber band etc.
 First major intelligence test which adequately sampled
 Test-retest reliability for some of these tests
total population of the United States
are near unacceptable
 Subtests labeled core, supplemental, or optional
Wechsler, Binet, and the Short Form
 Short form: test that has been abbreviated in length to reduce
time needed to administer, score and interpret
 used with caution, only for screening
 provide only estimates
 reducing the number of items usually reduces reliability and thus
validity
 Wechsler Abbreviated Scale of Intelligence
The Wechsler Test in Perspective
 Factor Analysis
o Exploratory factor analysis: summarizing data when
we are not sure how many factors are present in
our data
o Confirmatory factor analysis: used to test highly
specific factor analysis
CHAP.11: Other Individual Tests of Ability in Education and Special
Education Bayley Scales of Infants and Toddler Development – Third Edition (BSID-III)
- Base assessments on normative maturational developmental data
Alternative Individual Ability Tests Compared with the Binet and Wechsler - Designed for infants between 1 and 42mths
Scales - Assesses development across 5 domains: cognitive, language,
- None of these are clearly superior from a psychometric motor, socioemotional, and adaptive
standpoint - Motor scale: assumes that later mental functions depend on
- Some less stable, most more limited in their documented validity motor development
- Compare poorly to Binet and Wechsler on all accounts - Excellent standardization
- They don't rely on a verbal response as much as the B and W - Generally positive reviews
- Just use pointing or Yes/No responses, thus do not depend on the - Strong internal consistency
complex integration of visual and motor functioning - More validity studies needed
- Contain a performance scale or subscale - Widely used in research – children with Down syndrome,
- Their specificity often limits the range of functions or abilities that pervasive developmental disorders, cerebral palsy, language
they can measure impairment, etc
- Because they are designed for special populations, some - Most psychometrically sound test of its kind
alternatives can be administered totally without the - Predictive though?
verbal instructions Cattell Infant Intelligence Scale (CIIS)
- Based on normative developmental data
Specific Individual Ability Tests - Downward extension of Stanford-Binet scale for 2-30mth olds
- Earliest individual tests typically designed for specific purposes or - Similar to Gesell scale
populations - Rarely used today
- One of the first – Seguin Form Board Test – in 1800s – produced - Sample is primarily based on children of parents from lower and
only a single score middle classes and therefore does not represent the general
o Used primarily to evaluate mentally retarded population
adults and emphasized speed and performance - Unchanged for 60yrs
- After, the Healy-Fernald Test was developed as an exclusively - Psychometrically unsatisfactory
nonverbal test for adolescent delinquents
- Knox developed a battery of performance tests for non-English Major Tests for Young Children
adult immigrants to the US – administered without language; McCarthy Scales of Children’s Abilities (MSCA)
speed not emphasized - Measure ability in children between 2-8yrs
- These early individual tests designed for specific populations, - Present a carefully constructed individual test of human ability
produced a single score, and had nonverbal performance - Meager validity
scales - Produces a pattern of scores as well as a variety of composite
- Could be administered without visual instructions and used with scores
children as well as adults - General cognitive index (CGI): standard score with a mean of 100
Infant Scales and a standard deviation of 16
- Where mental retardation or developmental delays are o Index reflects how well the child has integrated prior
suspected, these tests can supplement observation, genetic learning experiences and adapted them to the
testing, and other medical procedures demands of the scales
Brazelton Neonatal Assessment Scale (BNAS) - Relatively good psychometric properties
- Individual test for infants between 3days and 4weeks - Reliability coefficients in the low .90s
- Purportedly provides an index of a newborn’s competence - In research studies
- Favorable reviews - Good validity? Good assessment tool
- Considerable research base Kaufman Assessment Battery for Children - Second Edition (KABC-II)
- Wide use as a research tool and as a diagnostic tool for special - Individual ability test for children between 3-18yrs
purposes - 18 subtests in 5 global scales called sequential processing,
- Commonly used scale for the assessment of neonates simultaneous processing, learning, planning, and knowledge
- Drawbacks: - Intended for psychological, clinical, minority-group, preschool,
o No norms are available and neuropsychological assessment as well as research
o More research is needed concerning the meaning and - Sequential-simultaneous distinction
implication of scores o Sequential processing refers to a child’s ability to solve
o Poorly documented predictive and construct validity problems by mentally arranging input in sequential or
o Test-retest reliability leaves much to be desired serial order
Gesell Developmental Schedules (GDS) o Simultaneous processing refers to a child’s ability to
- Infant intelligence measures synthesize info from mental wholes in order to solve a
- Used as a research tool by those interested in assessing infant problem
intellectual development after exposure to mercury, diagnoses of - Nonverbal measure of ability too
abnormal brain formation in utero and assessing infants with - Well constructed and psychometrically sound
autism - Not much evidence of (good) validity
- Children of 2.3mth to 6.3yrs - Poorer predictive validity for school achievement – smaller
- Obtains normative data concerning various stages in maturation differences between whites and minorities
- Individual’s developmental quotient (DQ) is determined - Test suffers from a noncorrespondence between its definition and
according to a test score, which is evaluated by assessing the its measurement of intelligence
presence or absence of behavior associated with maturation
- Provides an intelligence quotient like that of the Binet General Individual Ability Tests for Handicapped and Special Populations
o (development quotient / chronological age) x 100 Columbia Mental Maturity Scale – Third Edition (CMMS)
- But, falls short of acceptable psychometric standards - Purports to evaluate ability in normal and variously handicapped
- Standardization sample not representative of the population children from 3-12yrs
- No reliability or validity - Requires neither a verbal response nor fine motor skills
- Does appear to help uncover subtle deficits in infants
 - Requires subject to discriminate similarities and differences by Illinois Test of Psycholinguistic Abilities (ITPA-3)
indicating which drawing does not belong on a 6-by-9inch card - Assumes that failure to respond correctly to a stimulus can result
containing 3-5 drawings not only from a defective output system but also from a defective
- Multiple choice input or information-processing system
- Standardization sample is impressive - Stage 1: info must first be received by the senses before it can be
- Vulnerable to random error analyzed
- Reliable instrument that is useful in assessing ability in many - Stage 2: info is analyzed or processed
people with sensory, physical, or language handicaps - Stage 3: with processed info, individual must make a response
- Good screening device - Theorizes that the child may be impaired in one or more specific
Peabody Picture Vocabulary Test – Fourth Edition (PPVT-IV) sensory modalities
- 2-90yrs - 12 subtests that measure individual’s ability to receive visual,
- multiple choice tests that require subject to indicate Yes/No in auditory, or tactile input independently of processing and output
some manner factors
- Instructions administered aloud (not for the deaf) - purports to help isolate the specific site of a learning disability
- Purports to measure hearing or receptive vocabulary, presumably - For children 2-10yrs
providing a nonverbal estimate of verbal intelligence - Early versions hard to administer and no reliability or validity
- Can be done in 15mins, requires no reading ability - Now, with revisions, ITPA-3 psychometrically sound measure of
- Good reliability and validity children’s psycholinguistic abilities
- Should never be used as a substitute for a Wechsler or Binet IQ Woodcock-Johnson III
- Important component in a test battery or used as a screening - Evaluates learning disabilities
device - Designed as a broad-range individually administered test to be
- Easy to administer and useful for variety of groups used in educational settings
- BUT: Tendency to underestimate IQ scores, and problems - Assesses general intellectual ability, specific cognitive abilities,
inherent in the multiple-choice format are bad scholastic aptitude, oral language, and achievement
Leiter International Performance Scale – Revised (LIPS-R) - Based on the CHC three-stratum theory of intelligence
- Strictly a performance scale - Compares child’s score on cognitive ability with sore on
- Aims at providing a nonverbal alternative to the Stanford-Binet achievement – can evaluate possible learning problems
scale for 2-18yr olds - Relatively good psychometric properties
- For research, and clinical settings, where it is still widely utilized to - For learning disability tests, three conclusions seem warranted:
assess the intellectual function of children with pervasive o 1. Test constructors appear to be responding to the
developmental disorders same criticisms that led to changes in the Binet and
- Purports to provide a nonverbal measure of general Wechsler scales and ultimately to the development of
intelligence by sampling a wide variety of functions from the KABC
memory to nonverbal reasoning o 2. Much more empirical and theoretical research is
- Can be applied to the deaf and language-disabled needed
- Untimed o 3. Users or learning disabilities tests should take great
- Good validity pains to understand the weaknesses of these
Porteus Maze Test (PMT) procedures and not overinterpret results
- Popular but poorly standardized nonverbal performance measure Visiographic Tests
of intelligence - Require a subject to copy various designs
- Individual ability test Benton Visual Retention Test – Fifth Edition (BVRT-V)
- Consists of maze problems (12) - Tests for brain damage are based on the concept of psychological
- Administered without verbal instruction, thus used for a variety of deficit, in which a poor performance on a specific task is related to
special populations or caused by some underlying deficit
- Needs restandardization - Assumes that brain damage easily impairs visual memory ability
Testing Learning Disabilities - For individuals 8yrs+
- Major concept is that a child average in intelligence may fail - Consists of geometric designs briefly presented and then removed
in school because of a specific deficit or disability that - Computerized version developed
prevents learning Bender Visual Motor Gestalt Test (BVMGT)
- Federal law entitles every eligible child with a disability to a free - Consists of 9 geometric figures that the subject is imply asked to
appropriate public education and emphasizes special education copy
and related services designed to meet his or her unique needs - By 9yrs, any child of normal intelligence can copy the figures with
and prepare them for further education, employment, and only one or two errors
independent living - Errors occur for people whose mental age is less than 9,
- To qualify, child must have a disability and educational brain damage, nonverbal learning disabilities, emotional
performance affected by it problems
- Educators today can find other ways to determine when a child - Questionable reliability
needs extra help Memory-for-Designs (MFD) Test
- Processed called Response to Intervention (RTI): premise is that - Drawing test that involves perceptual-motor coordination
early intervening services can prevent academic failure for many - Used for people 8-60yrs
students with learning difficulties - Good split-half reliability
- Signs of learning problem: - Needs for validity documentation
o Disorganization - All these tests criticized because of their limitations in reliability
o Careless effort and validity documentation
o Forgetfulness - Good as screening devices though
o Refusal to do schoolwork or homework Creativity: Torrance Tests of Creative Thinking (TTCT)
o Slow performance - Measurement of creativity underdeveloped in psychological
o Poor attention testing
o Moodiness - Creativity: ability to be original, to combine known facts in new
ways, or to find new relationships between known facts
- Evaluating this a possible alternative to IQ
- Creativity tests in early stages of development
 - Torrance tests separately measure aspects of creative thinking - Avoid overinterpreting test scores
such as fluency, originality, and flexibility Be Especially Suspicious of Low Scores
- Does not meet the Binet and Wechsler scales in terms - Assume that subjects understand purpose of testing, want to succeed,
of standardization, reliability, or validity and are equally rested/free of stress
- Unbiased indicator of giftedness Consider Wide Discrepancies a Warning Signal
- Inconsistent tests, but available data reflect the tests’ merit and - May reflect emotional problems or severe stress
fine potential When in Doubt, Refer
Individual Achievement Tests: Wide Range Achievement Test-3 (WRAT-4) - With low scores, discrepancies, etc, refer the subject for individual
- Achievement tests measure what the person has actually acquired testing
or done with that potential - Get trained professional
- Discrepancies between IQ and achievement have traditionally Group Tests in the Schools: Kindergarten Through 12th Grade
been the main defining feature of a learning disability - Purpose of tests is to measure educational achievement in
- Most achievement tests are group tests schoolchildren
- WRAT-4 purportedly permits an estimate of grade-level Achievement Tests verses Aptitude Tests
functioning in word reading, spelling, math computation, and - Achievement tests attempt to assess what a person has learned
sentence comprehension following a specific course of instruction
- Used for children 5yrs+ o Evaluate the product of a course of training
- Easy to administer o Validity is determined primarily by content-related evidence
- Problems: - Aptitude tests attempt to evaluate a student’s potential for learning
o Inaccuracy in evaluating grade-level reading ability rather than how much a student has already learned
o Not proven as psychometrically sound o Evaluate effects of unknown and uncontrolled experiences
o Validity is judged primarily on its ability to predict future
CHAP: 12: Standardized Tests in Education, Civil Service, and the Military performance
- Intelligence test measures general ability
- When justifying the use of group standardized tests, test users often - These three tests are highly interrelated
have problems defining what exactly they are trying to predict, or what Group Achievement Tests
the test criterion is - Stanford Achievement Test one of the oldest of the standardized
Comparison of Group and Individual Ability Tests achievement tests widely used in school system
- Individual tests require a single examiner for a single subject - Well-normed and criterion-referenced, with psychometric
o Examiner provides instructions documentation
o Subject responds, examiner records response - Another one is the Metropolitan Achievement Test, which measures
o Examiner evaluates response achievement in reading by evaluating vocab, word recognition, and
o Examiner takes responsibility for eliciting a maximum reading comprehension
performance - Both of these are reliable and normed on big samples
o Scoring requires considerable skill Group Tests of Mental Abilities (Intelligence)
- Those who use the results of group tests must assume that the subject Kuhlmann-Anderson Test (KAT) – 8th Edition
was cooperative and motivated - KAT is a group intelligence test with 8 separate levels
o Many subjects tested at a time covering kindergarten through 12th grade
o Subjects record own responses - Items are primarily nonverbal at lower levels, requiring minimal reading
o Subjects not praised for responding and language ability
o Low scores on group tests often difficult to interpret - Suited to young children and those who might be handicapped in
o No safeguards following verbal procedures
Advantages of Individual Tests - Scores can be expressed in verbal, quantitative, and total scores
- Provide info beyond the test score - Scores at other levels can be expressed at percentile bands: like a
- Allow the examiner to observe behavior in a standard setting confidence interval; provides the range of percentiles that most likely
- Allow individualized interpretation of test scores represent a subject’s true score
Advantages of Group Tests - Good construction, standardization, and other excellent psychometric
- Are cost-efficient qualities
- Minimize professional time for administration and scoring - Good validity and reliability
- Require less examiner skill and training - Potential for use and adaptation for non-English-speaking individuals or
- Have more objective and more reliable scoring procedures even countries needs to be explored
- Have especially broad application Henmon-Nelson Test (H-NT)
Overview of Group Tests - Of mental abilities
Characteristics of Group Tests - 2 sets of norms available:
- Characterized as paper-and-pencil or booklet-and-pencil tests because o one based on raw score distributions by age, the other
only materials needed are a printed booklet of test items, a test on raw scores distributions by grade
manual, scoring key, answer sheet, and pencil - reliabilities in the .90s
- Computerized group testing becoming more popular - helps predict future academic success quickly
- Most group tests are multiple choice – some free response - does NOT consider multiple intelligences
- Group tests outnumber individual tests Cognitive Abilities Test (COGAT)
o One major difference is whether the test is primarily verbal, - Good reliability
nonverbal, or combination - Provides three separate scores though: verbal, quantitative, and
- Group test scores can be converted to a variety of units nonverbal
Selecting Group Tests - Item selection is superior to the H-NT in terms of selecting minority,
- Test user need never settle for anything but well-documented and culturally diverse, and economically disadvantaged children
psychometrically sound tests - Can be adopted for use outside the US
Using Group Tests - No cultural bias
- Reliable and well standardized as the best individual tests - Each of the subtests required 32-34 minutes of actual working time,
- Validity data for some group tests are weak/meager/contradictory which the manual recommends to be spread out over 2-3 days
Use Results with Caution - Standard age scores averaged some 15pts lower for African
- Never consider scores in isolation or as absolutes American students on the verbal battery and quantitative batteries
- Be careful using tests for prediction
Summary of K-12 Group Tests - Definite overall decline in verbal scores while quantitative and
- All are sound, viable instruments analytical scores are gradually rising

College Entrance Tests Miller Analogies Test

- SAT Reasoning Test, Cooperative School and College Ability Tests, and - Designed to measures scholastic aptitudes for graduate studies
American College Test - Strictly verbal
SAT Reasoning Test - 60 minutes
- Most widely used college entrance test - knowledge of specific content and a wide vocab are very useful
- Used for 1000+ private and public institutions - most important factors appear to be the ability to see relationships and
- Renorming of the SAT did not alter the standing of test takers relative a knowledge of the various ways analogies can be formed
to one another in terms of percentile rank - psychometric adequacy is reasonable
- New scoring (2400) is likely to reduce interpretation errors, as - does not predict research ability, creativity, and other factors
interpreters can no longer rely on comparisons with older versions important to grad school
- 45mins longer – 3hrs and 45mins to administer
- may disadvantage students with disabilities such as ADD The Law School Admission Test
- Verbal section now called “critical reading” – focus on reading - LSAT problems require almost no specific knowledge
comprehension - Extreme time pressure
- Math section eliminated much of the basic grammar school math - Three types of problems: reading comprehension, logical reasoning
questions (~half), and analytical reasoning
- Weakness: poor predictive power regarding the grades of students who - Weight given to the LSAT score is openly published for each school
score in the middle ranges approved by the American Bar Association
- Little doubt that the SAT predicts first-year college GPA - Entrance into schools based on weighted sum of score and GPA
o But, AfricanAmericans and Latinos tend to obtain lower - Psychometrically sound, reliability coefficients in the .90s
scores on average - Predicts first-year GPA in law school
o Women score lower on SAT but higher in GPA - Content validity is exceptional
- Bias for minority group members, as well as women
Cooperative School and College Ability Tests
- Falling out of favor Nonverbal Group Ability Tests
- Developed in 1955, not been updated Raven Progressive Matrices
- Purports to measure school-learned abilities as well as an individual’s - RPM one of the best known and most popular nonverbal group tests
potential to undertake additional schooling - Suitable anytime one needs an estimate of an individual’s general
- Psychometric documentation not strong intelligence
- Little empirical data support its major assumption – that previous - Groups or individuals, 5yrs-adults
success in acquiring school-learned abilities can predict future success - Used throughout the modern world
in acquiring such abilities - Uses matrices – nonverbal; with or without a time limit
- Research supports RPM as a measure of general intelligence, or
American College Test Spearman’s g
- Updated in 2005, particularly useful for non-native speakers of English - Appears to minimize the effects of language and culture
- Produces specific content scores and a composite - Tends to cut in half the selection bias that occurs with the Binet or
- Makes use of the Iowa Test of Educational Development Scale Wechsler
- Compares with the SAT in terms of predicting college GPA alone or in
conjunction with high-school GPA Goodenough-Harris Drawing Test (G-HDT)
- Internal consistency coefficients are not as strong in the ACT - Nonverbal intelligence test, group or individual
- Quick, east, and inexpensive
Graduate And Professional School Entrance Tests - Subject instructed to draw a picture of a whole an and to do the best
Graduate Record Examination Aptitude Test job possible
- GRE purports to measure general scholastic ability - Details get points
- Most frequently used in conjunction with GPA, letters of rec, and other - One can determine mental ages by comparing scores with those of the
academic factors normative sample
- General section with verbal and quantitative scores - Raw scores can be converted to standard scores with a mean of 100
- Third section which evaluates analytical reasoning – now essay format and SD of 15
- Contains an advanced section that measures achievement in at least 20 - Used extensively in test batteries
majors
- New 130-170 scoring scale The Culture Fair Intelligence Test
- Standard mean score of 500, and SD of 100 - Designed to provide an estimate of intelligence relatively free of
- Normative sample is relatively small cultural and language influences
- Psychometric adequacy is less than that of SAT – validity and reliability - Paper-and-pencil procedure that covers three age groups
- Predictive validity not great - Two parallel forms are available
- Overpredicts the achievement of younger students - Acceptable measure of fluid intelligence
while underpredicting performance of older students
- Many schools have developed their own norms and psychometric Standardized Tests Used in the US Civil Service System
documentation and can use the GRE to predict success in their - General Aptitude Test Battery (GATB) – reading ability test that
programs purportedly measures aptitude for a variety of occupations
- By looking at a GRE score in conjunction with GPA, graduate success o Makes employment decisions in govt agencies
can be predicted with greater accuracy than without the GRE o Attempts to measure wide range of aptitudes from general
- Graduate schools also frequently complain that grades no longer intelligence to manual dexterity
predict scholastic ability well because of grade inflation – the - Controversial because it used within-group norming prior to the
phenomenon of rising average college grades despite declines in passage of the Civil Rights Act of 1991
average SAT scores - Today, any kind of score adjustments through within-group norming in
o Led to corresponding restriction in the range of grades employment practices is strictly forbidden by law
- As the validity of grades and letters of rec becomes more questionable,
reliance on test scores increases
Standardized Tests in the US Military: The Armed Services Vocational
Aptitude Battery
- ASVAB administered to more than 1.3million people a year
- Designed for students in grades 11 and 12 and in postsecondary schools
- Yields scores used in both education and military settings
- Results can help identify students who potentially qualify for entry into
the military and can recommend assignment to various military
occupational training programs
- Great psychometric qualities
- Reliability coefficients are excellent
- Through computerized format, subjects can be tested adaptively,
meaning that the questions given each person can be based on his or
her unique ability
- This cuts testing time in half