PERSONNEL PSYCHOLOGY

1984. 37

INTRODUCING NEW METHODS FOR CONDUCTING

TRAINING EVALUATION AND FOR LINKING
TRAINING EVALUATION TO PROGRAM REDESIGN

J. KEVIN FORD
Michigan State University

STEVEN P. WROTEN
Ethyl Corporation

While training has been conceptualized as a continually evolving pro-

cess, the existing literature fails to provide adequate strategies for link-
ing training evaluation to training needs reassessment and program
redesign. This article presents two studies which describe
methodologies developed for a police recruit training program to ad-
dt^ss these deficiencies in the literature. In Study I. Lawshe's (1975)
Content Validity Ratio (CVR) approach was used to establish the job
relatedness of the content of the training pnsgrani. Results supported
the job relatedness of the training program as the average CVR value
was .27 (p < .05: A' = 105). In study 2. a new methodology, the
Matching Technique, was used to directly compare current training
emphasis with training needs. Results demonstrated a close match of
emphasis and training needs for the training program. One content
area of relative training deficiency was further examined to illustrate
the power of the Matching Technique for identifying areas requiring
" training redesign.

A useful model for describitig the systematic development and inter-

related components of a training program is based on an instructional
systems perspective (Goldstein, 1974; 1980). This training model in-
dicates that there should be a logical flow from the initial determination
of training needs to program development and to tt:aining evaluation.

This article is based on a paper presented at the 90th Annual Convention of the American
Psychological Association. August. 1982. Washington. DC.
The authors express their appreciation to Irwin Goldstein. Maiy D. Zaiesny. and Ken-
neth Wexley for helpful comments on earlier drafts of this article. Valuable research
assistance was provided by Sandra Miller. Vesta Martin. Ann Wilcox and Edith Garrcn.
Requests for reprints should be sent to J. Kevin Ford, Depanment of Psychology,
Michigan State University. East Lansing. MI 48824-1117.

© 1984 Personnel Psychology, Inc.

651
652 PERSONNEL PSYCHOLOGY

The training evaluation data are then used to systematically reassess train-
ing needs for possible program redesign. A key characteristic of the in-
structional system approach is the emphasis on the continuous use of
evaluative feedback to modify the existing training program. In other
words, training is seen as an evolving process which utilizes evaluative
information to adapt the program so that it better meets its stated objec-
tives (Goldstein and Buxton, 1982).
Despite the stated importance of training evaluation and the feedback
comjronent. few efforts have focused on content evaluation as an aspect
of training evaluation. Consequently, the existing training literature fails
to provide adequate strategies for evaluating existing training programs
in terms of the program's content and its job relatedness. Unfortunate-
ly, the training literature also fails to provide methods for linking train-
ing evaluation to training needs reassessment and program redesign.
This article describes two methodologies to address these deficien-
cies which were developed as part of a larger project to evaluate a recruit
training program for a large midwestem police force. The objectives
of the project were (1) to determine the extent to which the training content
domain is job related; and (2) to identify needed changes in training
content to improve its job relatedness. The methodologies developed
to meet these objectives are described below as two studies. Study 1
presents a unique application of the content validity ratio approach
(Lawshe. 1975) to evaluate the job relatedness of the police recruit training
program. The content validity strategy also provides the comprehen-
sive infomiation necessary for reassessing training needs. Study 2 presents
a new methodology called the Matching Technique which compares the
reassessed training needs with current training emphasis in order to link
training evaluation to training reassessment and program redesign.

Stt4dy I: Content Evaluation

In selection research, a considerable amount of effort has been devoted
to the concept of content validity and the development of job analysis
methods to determine the job relatedness of selection instruments. Un-
fortunately, a similar interest in systematically applying content valida-
tion strategies and job analysis methods for specific use in training con-
texts is sorely lacking (Prien, 1977). More specifically, Goldstein (1980)
noted in his recent review of the training literature that no procedure
currently exists which empirically evaluates the content validity of an
established training program.
Content validity, which is one aspect of the more general notion of
validity, refers to an evaluation of the representativeness or adequacy
of sampling the content domain of the job (El>el, 1977; Tenopyr, 1977).
In applying this notion to selection, the Principles for Validation
 FORD AND WROTEN 653

(American Psychological Association. 1980) and the Uniform Guidelines

(Equal Employment Opportunity Commission. 1978) state that selec-
tion procedures which purport to measure knowledges, skills and abilities
may justifiably use a content validation strategy if the knowledges and
skills are prerequisites for successful job performance. The same stan-
dards apply to the validation of training programs, since the objective
of training is to instill the knowledges, skills and abilities necessary for
effective job performance. A content validity strategy seems to be well
suited for determining the job relatedness of a training program.
The first step in this study's content validity strategy is to identify
the training content domain through a job element approach to job
analysis. This approach is normally applied to detemiine the content
validity of tests by directly matching test items to the job elements,
specified as the knowledges, skills, abilities and other personal
characteristics (KSAOs) that are necessary for job performance. When
applied to training, the job element approach specifies the content do-
main as consisting of those KSAOs taught or developed in the training
program.
Once the training content domain has been identified, the content can
be evaluated through a Content Validity Ratio (CVR) approach. The
CVR approach was initially introduced by Lawshe (1975) to aid the
development of content valid test items. To apply it to a training pro-
gram, the approach requires the use of "subject-matter experts"—people
who have a thorough knowledge of the job—to rate each training KSAO
on its importance or essentiality for job perfonnance. The two assump-
tions of the approach are that any KSAO perceived as important by more
than 50% of the experts has some degree of job relatedness, and that
as more experts beyond 50% perceive the KSAO as important, the greater
the degree of content validity.
A CVR is calculated by taking the number of experts who state that
a KSAO is important (A',), minus the number of experts who state that
a KSAO is not important (A'^, divided by the tota! number of experts

CVR =

A CVR is negative when fewer than half the experts say a KSAO is
important, positive, when more than half say it is important, and zero,
when exactly half the experts believe a KSAO is important to job per-
formance. A statistical test of significance for CVR values has been
developed by Schipper (personal communication. March, 1980). Addi-
tionally, an overall summary of the content validity of the training pro
654 PERSONNEL PSYCHOLOGY

gram can be generated by taking the mean of all the CVR values. This
summary statistic is called the Content Validity Index (CVI).

Method
Identification of Training Content Domain
The research was conducted for a large (over 1,100 personnel)
midwestem city police department. Each year, approximately 100 recmits
are trained by a staff of 15 to 20 police trainers. A six month. 800 hour
training program has been developed to prepare recruits for police patrol.
The training content was developed based on the perceived needs of
the training staff and administrative officials. To comprehensively identify
this training content domain. KSAOs taught or developed in the train-
ing program were derived from a rigorous three-step process structured
around a series of panel sessions and a large scale survey.
In the first step, three independent panels were created and held respon-
sible for generating a list of KSAOs that was an accurate and comprehen-
sive reflection of the training content. Two of the panels consisted of
training staff members and instructors, while the third panel consisted
of members of the most recently graduated recruit class. Each panel of
seven members met for three four-hour sessions.
Panel sessions were conducted according to a modified brainstorm-
ing approach (e.g.. see Primoff. 1975). Members were asked to generate
lists of KSAOs that were currently taught in the training program. The
process was loosely structured with panel members suggesting and
discussing relevant statements or topic areas. Midway through the ses-
sions, the training program syllabus and training materials were introduced
to stimulate discussion and insure comprehensiveness. A total of 963
KSAO statements were generated across the three panels by this process.
For the second step, the combined KSAO statements were categoriz-
ed into subgroups and duplicates were eliminated. To insure comprehen-
siveness and reduce ambiguities, a separate panel of training staff
members refined the list of suggested categories and edited the existing
statements. The revised list consisted of 393 KSAO statements.
For the third step, a survey containing the 393 KSAO statements was
developed to evaluate the accuracy and comprehensiveness of the train-
ing content domain. Participants who completed the survey included
15 training staff members and 35 police officers from the most recent
recmit class. Participants were asked to respond to each KSAO state-
ment in terms of whether or not the KSAO was taught or developed
in the training program. This process resulted in 10 KSAO statements
being eliminated. The content domain of the training program was thus
defined as consisting of 383 KSAOs.
 FORD AND WROTEN 655

48

40

32

Number
of 24
KSAO's

16

-.94 -.74 .54 -.34 .14 .06 | .26 .46 .66 .86
.17

Content validity ratio values^

"Content validity ratios > .17 are significant {p < .05; 105)

Figure 1. Frequency distributioD or content validity ratio values for the training
content domain.

Evaluation of Training Content Domain

The training content domain was evaluated for job relatedness through
the use of a Content Validity Ratio (CVR) approach. Three sources of
subject-matter experts were included for a total sample of 114. The ex-
perts included 64 experienced patrol officers who trained recruits on the
job and 31 patrol sergeants from the target city as well as 20 police per-
sonnel from similar sized communities outside the target city. The ex-
perts independently rated the importance of each KSAO to job perfor-
mance, using a seven-point scale ranging from no importance to ex-
treme importance. A decision rule was enacted whereby KSAOs with
656 PERSONNEL PSYCHOLOGY

TABLE 1
Overall Content Validity Indices (CVls) by Subgroup
Overall CVI Mitiimum CVI
Satiiple (383 KSAOs) for Significance
Target City .22 .19
Sample
(N = 94)
Outside City .49 .«
Sample
(N = 20)

Total
Combined .27 .17
Sample
(N =114)

ratings of above average importance or higher (five, six or seven) were

considered important, while KSAOs with ratings of average importance
or below (four or less) were considered relatively unimportant. The degree
of consensus among the experts was quantified by calculating a CVR
for each KSAO statement. The job relatedness of the total training pro-
gram was determined by calculating the mean of the CVR values, i.e.,
the Content Validity Index (CVI).

Results and Discussion

Figure 1 presents a frequency distribution which represents the number
of KSAO statements within intervals for various CVR values. The range
of CVR values extended from -.85 to +.93, with a modal response in
the .46 - .56 interval. KSAOs dealing with handling domestic distur-
bances, use of force issues and arrest situations had high CVR values.
Low CVR values were found for KSAOs that required some familiarity
for job performance, but were not essential for effective perfonnance
(e.g., knowledge of medical phrases and definitions). Of the 383 KSAOs
defining the training content domain, 237 (62%) had significant CVR
values (CVR > .\1\ p < .05).
Table 1 presents the CVI values for the target city sample of officers
and sergeants and the outside city sample. The CVIs for both samples
were significant and the CVI for the total combined sample of 114 was
.27 [p < .05). The intergroup correlations for the officer, sergeant and
outside city police rating groups indicated that all three groups had very
similar perceptions of how important the various job elements are to
job performance (r > .80, p < .001 for all comparisons).
To conclude, the high level of intergroup agreement supports the quality
 FORD AND WROTEN 657

of the experts judgments. The CVR and CVI values resulting from those
judgments supports the conclusion that the content of the training pro-
gram has a significant degree of job relatedness.

Study 2: Linking Evaluation to Redesign

Pressures to develop and implement a training program to address
perceived deficiencies in the workforce often prevent a thorough initial
analysis of training needs. Such a situation presents a critical need to
evaluate the existing program in order to prioritize training needs for
revision of the training program. Similarly, training programs established
through a strong initial needs assessment should periodically reassess
training needs for possible program nsvision. Unfortunately, systematic
attempts to translate training evaluation information into training needs
reassessment and to link that reassessment to program revision are lacking
in the training literature.
Some recent work has been completed by Macey and his associates
(Goldstein, Macey and Prien, 1982; Macey, 1982; Macey and Prien,
1981) on linking training needs assessment to the development of anew
training program. The methodology requires the specification of task
statements, job elements and critical incidents for the job. It relies on
the informed judgments of subject-matter experts who rate the tasks on
importance, frequency and difficulty of acquiring proficiency and rate
the job elements on importance, difficulty of learning and opportunity
to acquire on the job. The basic logic behind this approach is that the
training content should reflect those tasks and job elements that are most
important to the job, most difficult to leam, and have the least possibility
of being learned on the job. Training priorities are established by
calculating a composite index from the ratings of the task and job
elements. Critical incident data are then used to identify the specific
training content.
Similar to the Macey approach, this study attempts to prioritize train-
ing needs through the use of information from subject-matter experts.
Unlike Macey, this research focuses on an established training program
which has been evaluated for job relatedness through a Content Validi-
ty Ratio (CVR) approach (Study 1). The CVR values, based on ratings
of job importance, are used to reassess training needs. Importance ratings
of job elements have been found to be quite reliable across job types
(Macey and Prien, 1981) and to be more reliable than ratings of difficulty-
to-leam oropportunity-to-acquire. A methodology called the Matching
Technique is then used to link the training needs reassessment informa-
tion to program revision.
The new Matching Technique introduced here is conceptualized in
Figure 2 as a matrix in which training emphasis is directly compared
658 PERSONNEL PSYCHOLOGY

Low TRAINING EMPHASIS High

High

TRAINING
NEEDS

Low

Figure 2. Model of the matching technique.

to training needs. For general application, any measure of training needs

or emphasis can be used to apply the Matching Technique as long as
the measure reliably quantifies the extent to which areas need to be trained
or are emphasized in the training program.
The comparison of emphasis with needs identifies training "hits" and
"misses." Training hits refer to those content areas (defined in this study
as knowledges, skills and abilities) where the emphasis received in train-
ing appropriately reflects training needs.
Training misses are separated into two types: deficiencies and excesses.
Training deficiencies are content areas whose high training needs are
not matched by a high degree of emphasis in the training program. The
most extreme case would be a training omission, where a training need
is not emphasized at all in the training program. On the other hand, train-
ing excesses are areas receiving an excessive amount of emphasis relative
to their need to be trained.
An assumption behind the Matching Technique is that the greater the
number of misses (both deficiencies and excesses) uncovered, the less
effective the training program and the greater the need for refinement
or redesign. Generally, the number of deficiencies would be of more
critical concern than excesses. In addition, the Matching Technique not
only uncovers a need for refinement but also can provide the level of
 FORD AND WROTEN 659

specificity needed to indicate how the content of the program should

be changed to refine the program.

Method
Determining Training Needs
Since the CVRs from Study 1 indicated how important a KSAO is
in performing the job, the CVR values provided a reliable indication
of how much each KSAO needed to be trained. To reassess training
needs, the KSAO statements were divided into categories and sub-
categories by panels of training staff members to reflect the core cur-
riculum of the current training program. Seven major categories were
derived: (1) Laws; (2) Investigation; (3) Divisional Directives; (4) Courts;
(5) Physical Situations; (6) Communication; and (7) Human Relations.
In addition,the KSAOs in each category were further divided into from
3 to 7 subcategories. The " L a w s " category, for example, was com-
posed of the following six subcategories—Criminal, Traffic, Arrest, Rules
of Evidence, Liability, and Procedural Law. After dividing the KSAOs
in this manner, a Content Validity Index (CVI) was calculated for each
category and subcategory by averaging across the CVR values for the
appropriate KSAOs. In this way, the training need of each category and
subcategory was determined.

Training Emphasis and the Matching Technique

Each KSAO of the training content domain was rated by 35 recent
police recruits and 15 training staff members as to the emphasis placed
on each KSAO in the program. The ratings of training emphasis and
importance (completed by different rating groups) were highly correlated
(r = .68; p < .05) suggesting a large proportion of training hits or con-
firmations. The Matching Technique was then used to identify specific
areas for over- or under-emphasis that could not be uncovered using
correlational data.
To apply the Matching Technique, the training curriculum was divided
into the appropriate KSAO categories and subcategories. An objective
measure of training emphasis was detemiined by calculating the amount
of class time, in hours, devoted to each category and subcategory. The
training needs and training emphasis measures were then divided into
quartiles based on the range of CVI values and classroom hours, respec-
tively. Finally, the two dimensions of needs and emphasis were direct-
ly compared using the Matching Technique matrix (Figure 2) to uncover
training deficiencies and excesses.

Results and Discussion

As a first step in the matching process, the CVIs for the seven major
660 PERSOIWEL PSYCHOLOGY

TRAINING EMPHASIS**
Low High
0 50 100 150 200
High .5
1 5.7

.4

2
TRAINING
.3
NEEDS'^
6
.2

4 3
Low

•The numbers in the matrix indicate the following categories:

1 - Laws; 2 Investigation A Observation; 3 Divisional Directives;

4 = Courts; 5 - Physical Characteristics 8, Situations; 6 Com-
munication; 7 - Human & Community Relations

'draining emphasis as measured by classroom hours

*Training need as measured by content validity ratio values

Figure 3. Matching technique for KSAO categories*.

content categories were matched with the class time devoted to each
category to identify training excesses and deficiencies. The results of
matching training needs and emphasis are presented in Figure 3. The
matching matrix shows that the overall fit ofthe training time to police
officer training needs was quite good. This match supported the time
spent on the major content areas as well as supporting the conclusions
of job relevancy from Study 1. Ofthe training "misses." "Divisional
Directives" was a slight training excess while "Laws" was a slight train-
ing deficiency. Because training deficiencies are often more of a cen-
tral concern than excesses, the Laws category was further investigated
to determine the reason for its apparent deficiency.
To apply the Matching Technique at the subcategory level, training
time and CVI values were matched for the six Law subcategories
(Criminal, Traffic, Arrest, Evidence, Liability and Pn^cedural Law).
The results showed that four subcategories were training "hits'" or near
hits while the two subcategories of Evidence and Criminal Law were
training misses. Specifically, there was an apparent training excess for
 FORD AND WROTEN 661

TABLE 2
Content Validity Ratios (CVRs) for KSAO in the Evidence Law Suhcategorv

KSAO CVR*
• Knowledge of" what is or is not evidence, and what constitutes .78
it.s admissibility, relevancy and competency.
• Knowledge of who shall, when to. and how to legally obtain evidence .66
at the scene.
• Knowledge of how to maintain chain-of-cusiody for evidence. .58
• Knowledge of mles of evidence. .54
• Knowledge of "corpus delicta" and what evidence should be presented ,42
in association with the crime committed.
• Knowledge of different types of forms of evidence {such as. real, .19
circumstantial and direct evidence; declarations and confessions).
• Knowledge that a judge rules on relevancy and competency of evidence. .15
• Knowledge of privileged communications. -.07
• Knowledge of the legal rationale behind various rules of evidence. -.10
'CVRs > .17 are significant {p < .05; N - 105)

the Criminal Law subcategory and a training deficiency for the Evidence
subcategory.
To illustrate the power of the Matching Technique, the Evidence sub-
category was further examined to identify the specific knowledges and
skills that should be the focus of increased emphasis. Table 2 presents
the CVR values for the nine KSAOs grouped under ihe Evidence sub-
category. The higher CVR values (e.g.. Knowledge of how to maintain
chain-of-custody for evidence) suggest where the increased emphasis
should be placed to reduce the overall deficiency for the Evidence
subcategory.

General Discussion
This article has described two methods for evaluating the content validi-
ty of an established training program and for translating the training
evaluation information into training needs for program revision. The
research has provided a unique application of the Content Validity Ratio
approach to evaluate the job relatedness of the training program. In ad-
dition, the Matching Technique was presented to link training needs
reassessment and program revision.
The two methods utilized in these studies successfully met the objec-
tives of the f)olice department to evaluate the job relatedness of the recmit
training program and to suggest needed changes. The results of the Con-
tent Validity Ratio approach demonstrated that the content of the train-
ing program was job related. The Matching Technique highlighted the
close match of training needs and training emphasis at a general, category
level of analysis. Further investigation suggested minor changes in the
emphasis placed on various content areas.
662 PERSONNEL PSYCHOLOGY

Applications
The methods used in the present research have been applied to other
settings and for other purposes. For example, the CVR approach has
been used to develop job-related behavioral rating scales for entry level
psychiatric aides {Distefano, Pryer, and Erffmeyer. 1983) and to evaluate
the validity of the skill dimensions and exercise components of an assess-
ment center for secondary school principals (Schmitt and Noe, 1983).
The Matching Technique has recently been applied to customize a
company-wide interpersonal skills training program in a large
petrochemical company (see Ford and Wroten, 1982; Wroten, 1983).
In this case, the training needs of supervisors at individual plant loca-
tions were matched with the training emphasis of a generic, company-
wide skills training program. Adjustments in the generic training pro-
gram were then made for each individual plant location based on the
identification of what training areas should receive more (or less) em-
phasis at that location. The decision to customize the generic training
program was further aided by a dollar impact analysis of the interper-
sonal skill dimensions (Wroten. 1983). This utility analysis helped iden-
tify which of the training deficiencies should be emphasized more in
the customized training program.
In another area of application, the information gathered from the mat-
ching process can profitably be used to improve the job relatedness of
the testing procedures of the training program. In the police training
program, the major form of recmit evaluation is through the use of multi-
ple choice type examinations of major category and subcategory con-
tent areas. The job relevance of these tests can be determined by ex-
amining the importance (i.e., CVR or CVI values) of the knowledges
tapped by the exam items. The training staff can then revise the tests
to insure that the test items reflect the more important knowledges needed
for job performance, For example, a test on Laws should emphasize
items which tap KSAOs relevant to Arrest (CVI = .73) more so than
other Law-related areas such as Traffic Law (CVI = . 19). Additional-
ly, the Evidence Law exam can be examined to insure that the knowledge
items with high CVR values are emphasized on the test while low CVR
value knowledge items are minimized on the test. In addition, the specific
identification of important sidlls through the CVR approach provides
the information needed to begin building more work-sample oriented
testing procedures.

Enhancements and Limitations

The two studies presented do have certain limitations. The results of
the studies depend to a large extent on the validity of judgments made
by job "experts." While experts are knowledgeable of what they are
 FORD AND WROTEN 663

rating, this knowledge may impair their objectivity. Cordner (1980),

for example, has cautioned that job analyses completed by police per-
sonnel may only serve to reinforce and legitimize the status quo and
police-oriented values. From this perspective, the high degree of inter-
rater agreement among independent samples of police personnel may
be more indicative of this pervasive orientation than an indication of
the "quality" ofthe data. Therefore, while the training content was
basically confirmed, the responses ofthe raters may be failing to pro-
vide the critical inputs necessary to initiate major changes in training
content. We attempted to counter such possibilities by including as many
individuals from various levels ofthe organization in the panel and rating
sessions as possible. One suggestion for enhancing the development of
critical inputs for change would be to gather ratings of importance of
the job content domain from sources other than police personnel (e.g.,
citizen groups, city administrators).
Other research decisions add to the judgmental characteristic of the
studies. One such decision was the conversion of the seven-point job
relatedness scale to Lawshe's dichotomous scale for determining the CVR
value—i.e., points 5, 6 and 7 were combined as indicating that the KSAO
was essential to the job. Although this seemed reasonable, and even
conservative, based on the point 5 anchor (KSAO is "above average
in importance" for performing the job), a cutoff one point higher or
lower would obviously have affected the validity results.
Another issue concerned the level of specificity and the comprehen-
siveness of the KSAOs. The amount of detail could not be too over-
burdening, but the KSAOs also could not be so broad and general that
their usefulness for determining job relatedness and training needs would
be diminished. The researchers opted to err more toward the former situa-
tion than the latter. To ensure comprehensiveness, the researchers again
relied upon input from several independent sources. These options,
although not guaranteeing, do go a long way toward collecting the most
useful data. Unfortunately, such options also proved to be more time-
and people-consuming.
As a final caution, the results of the Matching Technique must be
interpreted in relationship to how training needs and training emphasis
were operationalized. For example, training emphasis, as operationalized
in the present research as "time spent," provided the police department
with sufficient information to make program redesign decisions.
However, it must be realized that the emphasis measure did not directly
take into account that an inordinate amount of time may be required
to train certain KSAOs compared to their job relatedness. Although some
KSAOs may have low CVR values, they may still be important because
they are much more difficult to leam or are prerequisites to learning
664 PERSONNEL PSYCHOLOGY

subsequent KSAOs which are important for the job. One alternative to
enhance the construct validity of the training emphasis measure would
be to have job experts rate the job elements on difficulty to leam and
incorporate that information with the previous results. Unfortunately,
Macey and Prien (1981) report that such rating scales usually have low
reliability. A second alternative is to conduct panel sessions with train-
ing experts. The panel sessions would concentrate on reaching a con-
sensus on the leaming difficulty of those KSAOs in the categories or
subcategories identified through the Matching Technique as being defi-
cient or excessive.
Even recognizing these possible limitations, the application of the CVR
approach and the development of the Matching Technique proved quite
useful for validating and redesigning a police recmit training program.
More research efforts are needed to develop strategies which provide
the,feedback linkages necessary to modify existing training programs
and to insure that the programs are better meeting their stated objectives.

REFERENCES

American Psychological Association. Division of Industrial/Organizational Psychology

{1980). Principles for the Validation and Use of Personnel Selection Procedures,
(2nd e d ) . Berkeley, CA: Author.
Cordner, G. W. (1980). Job analysis and the police: Benefits and limitations. Journal
of Police Science and Administration. 355-362.
Distefano. M, K.. Ptyer. M, W., and Erffmeyer. R. C. (1983). Application ol content
validity methods to the development of a job related performance rating criterion.
PERSONNEL PSYCHOUXIY. 36. 621-632.
Ebel. R. L. (1977). Prediction? Validation? Construct Validity? PERSONNEL
PSYCHOLOGY, 30, 55-63.
Equal Employment Opportunity Commission. Department of Labor and Department of
Justice. (1978). Uniform guidelines on employee selection pnKcdures. Federal
Register. 43(\66). 38290-38315.
Ford. J. K.. and Wroten. S. P. (1982). A content validity ratio approach to determine
training needs. Presented at the 90th convention of the American Psychological
Association, Washington, DC.
Goldstein. 1. L. (1974). Training: Program Development and Evaluation. Monterey.
CA: Brooks/Cole.
Goldstein, 1. L. (1980). Training in work organizations. Annual Review of Psychology,
31. 229-272.
Goldstein. 1. L.. and Buxton, V. M. (1982). Training and human performance. In E.
A. Fleishman (Ed.). Human Performance and Productivity, (Vol. I). Hiltsdale.
NJ: Erlbaum.
Goldstein, 1. L., Macey. W. H.. and Prien. E. P. (1981). Needs assessment approaches
for training development. In H. Meltzer and W. R. Nord (Eds.). Making Organiza-
tions More Productive. New York: Wiley.
Lawshe, C. H. (1975). A quantitative approach to content validity. PERSONNEL
PSYCHOLOGY. 28, 563-575.
 FORD AND WROTEN 665

Macey. W. H. (1982). Linking training needs a.ssessment lo training program de.fign.

Presented at the 90th Convention of the American Psychological Association,
Washington, DC.
Macey, W. H.. and Prien, E. P. (1981). Needs assessment: Program and individual
development. Presented ai ihe 89th convention of the American Psychological
Association, Los Angeles, CA.
Prien. E. P. (1977). The function of job analysis in content validation. PERSONNEL
PSYCHOLOGY. .?(J. 167-174.
Primoff. E. S. (1975). How to prepare and conduct job element examinations. (TS-75-1)
Washington, DC: U.S. Government Printing Office.
Schmiti. N.. and Noe. R. A. (1983). Demonstration of content validity: An assessment
center example. Presented at the 91st Convention of the American Psychological
Association, Anaheim. CA.
Tenopyr, M. L. (f977). Content-construct confusion. PERSONNEL PSYCHOLOGY, 30,
47-54.
Wroten, S. P. (1983). Study of Supervisor TIPS Workshops. Technical Report. Ethyl
Corporation, Richmond, VA.