Developing Occupational

Questionnaires
 Agenda

• Background and Context

• Steps in Developing Occupational Questionnaires

• Implementation Issues

2
Background and Context
 Impact of Automation

• Has drastically changed how agencies recruit,

receive applications, and assess and select
candidates

• Has made it easy for applicants to apply

• Challenge: Paring down the number of candidates

while making consistent, accurate and merit-
based distinctions among them

4
 Current Hiring Context

• Need for high quality hires to carry out Federal

agency missions

• High applicant volume

• Timeliness goals

• Lack of assessment and measurement expertise

at many agencies

5
What is an Occupational Questionnaire?

• Assessment method used

to screen, rate, and rank
job applicants
• Commonly delivered
through automated
staffing systems used for
Federal hiring
• Consists of self-ratings of
training and experience
(T&E)

6
Benefits of Occupational Questionnaires

• Positions can be filled quickly

• Relatively inexpensive and efficient
• Low burden and high face validity for applicants
• Wide variety of KSAs/competencies can
be assessed
• Easy to automate
• Test security is not an issue
• Occupational Questionnaires are familiar to
agencies

7
Challenges of Occupational Questionnaires

• Response inflation/deflation

• Lower validity

• Less differentiation among candidates

• Less suitable for entry-level positions

• Requires adequate time to develop

• Subject matter expert (SME) involvement is

needed
8
 Setting Up for Success

1. Quality job analysis information

2. Trained and experienced occupational
questionnaire developers
3. SME input and review
4. Infrastructure that promotes quality and
accountability

9
 Steps in Developing
Occupational Questionnaires
 Topics Covered

1. Developing an assessment plan

2. Assessing minimum qualifications

3. Writing items

4. Selecting rating scales

11
1. Developing an Assessment Plan

• Identify content areas

(KSAs/competencies &
tasks)

• Identify a target weight for

each content area

• Set a target for the total

number of items

• Map the total number of

items
12
1. Developing an Assessment Plan

Step 1: Identify content areas

• Review job analysis data to select competencies
(and supporting tasks) by reviewing scores for:
• Importance – Higher ratings indicate greater
importance
• Required at entry – Lower ratings indicate the
competency is needed at entry (not learned on
the job)
• Distinguishing value – Higher ratings indicate
the competency is valuable for distinguishing
superior from fully successful employees

13
1. Developing an Assessment Plan

Step 1: Identify content areas, continued

• Measurable - Not all competencies (e.g.,
Integrity/Honesty) are best measured with a self-
report instrument
• Unidimensional – Only measure one competency
at a time (e.g., measure Oral Communication and
Written Communication separately)
• Job domain coverage – Try to capture the full
range of general and technical job requirements

14
1. Developing an Assessment Plan

Step 2: Identify target weight for each content area

• Weights should equal 100%
• Recommend equal weights for each content area
• Use job analysis or subject matter expert input to
determine if any areas are more important and
should be given more weight

15
1. Developing an Assessment Plan

Step 3: Set a target for the total number of items to

assess
• Consider the following:
• Number of content areas that need to be
covered
• Complexity of the content areas
• Applicant burden
• General guidelines:
• No fewer than 10 items
• No more than 40 items
• 40 items take approximately 20 minutes to
complete
16
1. Developing an Assessment Plan

Step 4: Map the total number of items

• Determine the total number of items per content
area
• Issues to consider:
• Complexity of the content areas
• Target weight for each content area

17
1. Developing an Assessment Plan

Sample Completed Assessment Plan

Note: These numbers may change as you begin creating the questionnaire.

HR Specialist GS-11
Target
Content Area # of Items % of Items Weight
Content Area 1 3 20% 20%
Content Area 2 3 20% 20%
Content Area 3 3 20% 20%
Content Area 4 3 20% 20%
Content Area 5 3 20% 20%

Total 15 100%

18
2. Assessing Minimum Qualifications

• Use Yes/No or multiple choice questions

• Use clear, unambiguous language that any
applicant can understand
• Include response options for both qualified and
unqualified applicants
• Describe experience requirements at each grade
level identified, such that distinctions between
grade levels are clear
• Describe education requirements, including
references as appropriate
• Provide a link to the Qualification Standard and/or
Individual Occupational Requirements (IOR) as
appropriate
19
 3. Writing Items

• Goal is to write the item in terms of specific,

observable, and verifiable behaviors
• Crafting quality items requires time, effort, and
practice
• Pick a strong action verb
• Writes vs. Develops
• Assembles vs. Prepares
• Should be written so that a colleague or
supervisor could verify that the applicant
performed that behavior

20
 3. Writing Items

Follow the Formula:

Perform To Whom or To Produce What?

What? + What? + OR Why? How?
(Action Verb) (Direct Object) (Qualifying
Statement)

Examples:
Sort + incoming mail + into functional groups
for distribution.

Proofread and edit + letters, memos, e-mail or

other written communication documents +
to address format or grammatical, spelling, or
typographical errors.
21
 3. Writing Items

Item writing tips

• Write items in behavioral, observable, and
verifiable terms
• Write clearly and succinctly
• Develop items that meaningfully distinguish
among applicants
• Keep your competencies unidimensional
• Consider the level of job-specific experience
required for the position
• Emphasize quality over quantity

22
 3. Writing Items: Pitfalls

Unnecessary words add to the length of an item,

do not add meaning, and can actually make the task
seem more complicated that it is. You want your
items to be concise so they are easily understood.

Relay various types of written information and

communications, both in e-mail and memo format, to
Wordy:
individuals at all levels of the organization and on
multi and varied topics.
Write correspondence (e.g., e-mails, memos) on
More
varied topics for distribution to individuals at all
Concise:
organizational levels.

23
 3. Writing Items: Pitfalls

Double-Barreled items include multiple tasks or

competencies, thus forcing applicants to choose
which part of the statement they will respond to,
giving you incomplete or ambiguous information as
to what experience the applicant has.
Assemble quarterly performance data from internal
Double-
office sources into a computer spreadsheet and
Barreled:
prepare a written report.
Assemble performance data (e.g., quarterly,
monthly) from internal office sources into a
Two computer spreadsheet.
Statements:
Write a report summarizing trends in performance
data for internal decision making purposes.

24
 3. Writing Items: Pitfalls

Overly specialized items, or items based on job-

specific task statements, may unnecessarily omit
qualified applicants. Although there may be certain
specialized or technical expertise that is required,
experience in other job settings may be equally as
valuable, particularly when assessing general
competencies.

Overly Add, subtract, multiply, divide, and compute

Specific: percentages to calculate pay, time, and benefits.
Appropriate Use addition, subtraction, multiplication, and
Level of division, and compute percentages to perform basic
Generality: calculations.

25
 3. Writing Items: Pitfalls

Vague, subjective, or evaluative wording can

cause the applicant to incorrectly interpret the tasks
or statements. Statements that are clear and
specific allow applicants to more accurately rate
themselves.

Perform all required quality control procedures

Evaluative
on time and attendance reports in an effective
Wording:
and thorough manner.
Verify the accuracy of data or information in a
Behavioral: report or database following established
procedures.

26
 3. Writing Items: Pitfalls

Abbreviations/Acronyms may not be understood

or hold the same meaning for all applicants. It’s
better to spell out all abbreviations and acronyms to
ensure the item is interpreted the same way by all
applicants.

Abbreviated: Prepare MOUs to secure new contracts.

Prepare proposals or Memorandums of
Written Out: Understanding in order to secure new
contracts.

27
 4. Selecting a Rating Scale

• Types of Scales
• Generic vs. Customized
• Pros and Cons

• Various Formats

• Additional Tips

28
 4. Selecting a Rating Scale

Generic Rating Scales

• Can be used across items regardless of the
content area being assessed or the position
being filled.

Customized Rating Scales

• Are specific to each occupation, grade level,
and/or item.

29
 4. Selecting a Rating Scale

Generic Rating Scales

Pros
• Relatively inexpensive and efficient
• Measurement expertise is not required
• Positions can be filled quickly
• Can be used across items
(regardless of content area or position)

30
 4. Selecting a Rating Scale

Generic Rating Scales

Cons
• Experience-based scales may result in less
discrimination among candidates for entry- and
high-level positions
• Experience-based scales may be less appropriate
for entry-level positions
• Transparency in terms of the ‘best’ answer
• May result in lower level of applicant engagement
in responding

31
 4. Selecting a Rating Scale

Customized Scales
Pros
• Response alternatives are specific to each
occupation and grade level (increases face
validity for applicants)
• SME involvement leads to greater question
validity and upper management level buy-in
• Greater applicant differentiation
• Can be developed so that they are less
transparent to applicants
• Higher level of applicant engagement

32
 4. Selecting a Rating Scale

Customized Scales
Cons
• Response alternatives are specific to occupation
and grade level
• Response alternatives are more difficult and time
consuming to develop
• Requires measurement experts to develop the
items
• Requires extensive SME involvement (dependent
upon availability, interest, and competence of
SME)
• Requires periodic review to ensure the questions
are still applicable to occupation 33
 4. Selecting a Rating Scale

Various Formats

• Yes/No

• Multiple Choice

• Forced Choice (Ranking)

• Select All That Apply

34
 4. Selecting a Rating Scale

Additional Tips:
• Goal: Help applicants be as accurate as possible
when making their self-report ratings
• Ensure that items and rating scales are focused
on clearly stated, observable and verifiable
behaviors
• Use specific and unambiguous language
• Use rating scales that are anchored by clear,
verifiable, and behaviorally based descriptions of
what each response option means

35
Implementation Issues
 Design a Verification Process

• Manual verification of
supporting materials by
trained HR professionals
can provide a valuable
check on the accuracy of
self-report ratings
• Minimum qualifications
• Education
• Ensure a standardized
and consistent process

37
 Design a Verification Process

• Consider use of follow-up assessments (e.g.,

structured interview, work sample, reference
check) to verify competencies of top applicants
• Notify applicants in advance of agency verification
policy, as well as use of follow up assessments
as appropriate
• For example: ‘If, after reviewing your resume
and/or supporting documentation, a
determination is made that you have
overstated your qualifications and/or
experience, you will be removed from
consideration or your score will be lowered.’

38
 Sample Verification Statement

I certify that, to the best of my knowledge and belief, all of the

information included in this questionnaire is true, correct, and provided
in good faith. I understand that if I make an intentional false statement,
or commit deception or fraud in this application and its supporting
materials, or in any document or interview associated with the
examination process, I may be fined or imprisoned (18 U.S.C. 1001),
my eligibilities may be cancelled, I may be denied an appointment, or I
may be removed and debarred from Federal service (5 CFR part 731).
I understand that any information I give may be investigated. I
understand that responding “No” to this item will result in my not being
considered for this position.

A. Yes, I certify that the information provided in this questionnaire is

true, correct and provided in good faith, and I understand the
information provided above.
B. No, I do not certify the information provided in this questionnaire is
true, correct and provided in good faith.
C. No, I do not understand the information provided above. 39
 Evaluating the Process

1. Were the items on the occupational

questionnaire clear and easily understood by all
applicants?
• Were applicants confused by any of the items,
rating scales, or response options? Can you
improve the questionnaire based on the
questions you received?
2. Did the occupational questionnaire items help to
make meaningful distinctions among applicants?
• Are the scores spread out or clumped
together? Are too many scores in the high or
low range?

40
Evaluating the Process, Continued

3. Did scores on the occupational questionnaire

help to identify the most highly qualified
applicants?
• Obtain feedback from the HR Specialist who
reviewed the applications and from the hiring
manager.
4. Is there a high return-on-investment associated
with use of the occupational questionnaire?
• Do the benefits of using the questionnaire
(e.g., better-quality applicants) outweigh the
cost of developing and implementing it?

41
 Infrastructure

• Establish clear standards of quality

• Train HR professionals in good assessment and
item writing practices
• Arrange for SME involvement in the development
and review of every occupational questionnaire
• Create an environment that promotes quality
and accountability
• Centralize or designate certain individuals to
develop occupational questionnaires
• Evaluate results

42
 Red Flags

• Low score variability

• Indicates applicants are answering the
question the same, negating the purpose of the
questionnaire
• Returned certifications
• No selection is made
• Hiring manager dissatisfaction
• High volume of applicant inquiries

43
 One Final Thought

“The ultimate goal of staffing is to make good hires

as efficiently as possible. It is not to make bad
hires quickly.”
- Maximizing the Value of Your Investment
in Qualifications Screening Technology.
Rocket-Hire. Feb 2004.

44
 Contact Information:

[email protected]