18/03/2025

WHAT IS UTILITY?

• usefulness or practical value of testing to improve efficiency

Factors That Affect a Test’s Utility

UTILITY  Psychometric soundness
 reliability and validity of a test
Lesson 11, Psychological Assessment  index of utility can tell us something about the PRACTICAL
VALUE of the information derived from scores on the test
 Test scores are said to have utility if their use in a particular
situation helps us to make better decisions

WHAT IS UTILITY? WHAT IS UTILITY?

Factors That Affect a Test’s Utility Factors That Affect a Test’s Utility
 Psychometric soundness  Cost
A test may be PSYCHOMETRICALLY SOUND, it may have  economic, financial, or budget-related
little utility— particularly if the targeted testtakers demonstrate a one of the most basic elements in any utility analysis is the
tendency to “bend, fold, spindle, mutilate, destroy, tamper FINANCIAL COST of the selection device
with,” or otherwise fail to scrupulously follow the test’s directions.
COST in the context of test utility refers to disadvantages,
losses, or expenses in both economic and noneconomic terms

1
 18/03/2025

WHAT IS UTILITY? WHAT IS UTILITY?

Factors That Affect a Test’s Utility

Factors That Affect a Test’s Utility  Cost
 Cost (1) payment to professional personnel and staff associated with test
 expenditures associated with testing or not testing administration, scoring, and interpretation, (2) facility rental,
mortgage, and/or other charges related to the usage of the test
 allocate funds to purchase (1) a particular test, (2) a supply of blank facility, and (3) insurance, legal, accounting, licensing, and other
test protocols, and (3) computerized test processing, scoring, and routine costs of doing business
interpretation

WHAT IS UTILITY? WHAT IS UTILITY?

Factors That Affect a Test’s Utility Factors That Affect a Test’s Utility
 Cost  Benefits
Non Economic Costs: pain, suffering, inconvenience, emotional whether the benefits of testing justify the costs of administering,
distress, loss of companionship,loss of enjoyment of life scoring, and interpreting the test
BENEFIT refers to profits, gains, or advantages
thoughtfully designed and well-run testing programs
a) an increase in the quality of workers’ performance;

2
 18/03/2025

WHAT IS UTILITY? WHAT IS UTILITY ANALYSIS?

Factors That Affect a Test’s Utility  UTILITY ANALYSIS may be broadly defined as a family of
 Benefits techniques that entail a COST–BENEFIT ANALYSIS designed
to yield information relevant to a decision about the usefulness
b) an increase in the quantity of workers’ performance
and/or practical value of a tool of assessment
c) a decrease in the time needed to train workers
d) a reduction in the number of accidents
 UTILITY ANALYSIS may be undertaken for the purpose of
e) a reduction in worker turnover. evaluating whether the benefits of using a test (or training program
Admission Test: Graduates or intervention) outweigh the costs.
Diagnostic Test: Responsive Interventions

WHAT IS UTILITY ANALYSIS? WHAT IS UTILITY ANALYSIS?

If undertaken to evaluate a test, the utility analysis will help make If undertaken to evaluate a test, the utility analysis will help make
decisions regarding whether: decisions regarding whether:

one test is preferable to another test for use for a specific purpose the addition of one or more tests (or other tools of assessment) to
one or more tests (or other tools of assessment) that are already in
use is preferable for a specific purpose
one tool of assessment (such as a test) is preferable to another tool
of assessment (such as behavioral observation) for a specific purpos
no testing or assessment is preferable to any testing or assessment

3
 18/03/2025

HOW IS A UTILITY ANALYSIS HOW IS A UTILITY ANALYSIS

CONDUCTED? CONDUCTED?

 EXPECTANCY DATA
EXAMPLE: A school psychologist uses an expectancy table to
 EXPECTANCY DATA determine the likelihood that students who score within a particular
An expectancy table can provide an indication of the likelihood range on an aptitude test will succeed in regular classes as opposed
that a testtaker will score within some interval of scores on a to special education classes.
criterion measure—an interval that may be categorized as “passing,”
“acceptable,” or “failing.” Likelihood that individuals who score within a given range on the
predictor will perform successfully on the criterion

HOW IS A UTILITY ANALYSIS HOW IS A UTILITY ANALYSIS

CONDUCTED? CONDUCTED?

 EXPECTANCY DATA  TAYLOR-RUSSELL TABLES

Taylor-Russell tables provide an estimate of the extent to which
inclusion of a particular test in the selection system will
improve selection.

New Test Existing Test or Tool Improved Selection

4
 18/03/2025

HOW IS A UTILITY ANALYSIS HOW IS A UTILITY ANALYSIS

CONDUCTED? CONDUCTED?

Original Base Rate of .60, meaning that 60% of those hired under the
existing system are successful in their work.
New Test: validity coefficient of .55
 TAYLOR-RUSSELL TABLES Selection Ratio: 0.20
provide an estimate of the percentage of employees hired by New Base Rate: 0.88.
the use of a particular test who will be successful at their jobs, Selection Ratio: 5% or 0.05
(the test’s validity, the selection ratio used, and the base rate) New Tests: validity coefficient of .15
New Base Rate: ???

THE CUT SCORE IN USE

 CUT SCORE as a (usually numerical) reference point derived

as a result of a judgment and used to divide a set of data into
two or more classifications.

5
 18/03/2025

THE CUT SCORE IN USE THE CUT SCORE IN USE

 RELATIVE CUT SCORE (NORM-REFERENCED CUT

SCORE) may be defined as a reference point—in a distribution of
test scores used to divide a set of data into two or more  FIXED CUT SCORE is a pre-determined, minimum
classifications—that is set based on norm-related passing score for a test or assessment, independent of the
considerations rather than on the relationship of test scores to a performance of other test-takers, used to determine whether a
criterion. candidate passes or fails.

THE CUT SCORE IN USE THE CUT SCORE IN USE

 MULTIPLE CUT SCORES refer to the use of two or more cut  MULTISTAGE (OR MULTIPLE HURDLE) method
scores with reference to one predictor for the purpose of involves a series of sequential selection steps, where candidates
categorizing testtakers. must pass each "hurdle" (or stage) to advance to the next, with
failure at any stage resulting in disqualification.
 class member will be assigned to one category (e.g.,A, B, C, D, or F)

6
 18/03/2025

YES = 1
METHODS FOR SETTING CUT SCORES METHODS FOR SETTING CUT SCORES
NO = 0

THE ANGOFF METHOD THE ANGOFF METHOD

 setting fixed cut scores can be applied to personnel selection TASKS  the judgments of the experts are averaged to yield cut scores for
as well as to QUESTIONS regarding the presence or absence of a the test
particular trait, attribute, or ability Item SME 1 SME 2 SME 3
Number
relating to the determination of whether or not testtakers possess 1 1 0 1 2/3 = 0.66
a particular trait, attribute, or ability, an EXPERT PANEL MAKES 2 1 1 1 3/3 = 1.00
3 0 0 1 1/3 = 0.33
JUDGMENTS concerning the way a person with that trait,
4 0 0 0 0/3 = 0.00
attribute, or ability would respond to test items.
5 1 1 1 3/3 = 1.00
3/5 = 0.60 2/5 = 0.40 4/5 = 0.80 9/15 = 0.60

METHODS FOR SETTING CUT SCORES METHODS FOR SETTING CUT SCORES

THE KNOWN GROUPS METHOD THE KNOWN GROUPS METHOD

administered to all incoming freshmen
 Collection of data on the predictor of interest from groups
known to possess, and not to possess, a trait, attribute, or Analyzes the scores: a group who passed the course and
ability of interest. earned credit, and a group who did not earn credit for the
 “Who Needs to Take Remedial Math? Test” (WNTRMT) course because their final grade was a D or an F
setting a cut score based on the performance of two score at the point of least difference between the two
contrasting groups: (1) students who successfully completed groups
college-level Math, and (2) students who failed college-level
math

7
 18/03/2025

METHODS FOR SETTING CUT SCORES METHODS FOR SETTING CUT SCORES

IRT-BASED METHODS

 IRT framework, each item is associated with a particular

level of difficulty
 “Pass” --- must answer items (above minimum level of
difficulty,) --- experts and serves as the cut score

METHODS FOR SETTING CUT SCORES METHODS FOR SETTING CUT SCORES

ITEM-MAPPING METHOD BOOKMARK METHOD

 arrangement of items in a histogram  begins with the training of experts with regard to the
 judges who have been trained regarding minimal minimal knowledge, skills, and/or abilities that testtakers
competence required for licensure are presented with should possess in order to “pass”
sample items from each column
 asked whether or not a minimally competent licensed  the experts are given a book of items, with one item
individual would answer those items correctly about half the printed per page, such that items are arranged in an
time ascending order of difficulty

8
 18/03/2025

METHODS FOR SETTING CUT SCORES

BOOKMARK METHOD
 the expert then places a “bookmark” between the two
pages (or, the two items) that are deemed to separate
testtakers who have acquired the minimal knowledge, skills,
and/or abilities from those who have not.

 Additional rounds of bookmarking with the same or other

judges may take place as necessary.