How to design examination test items that are credible, valid and reliable

Pre-Amble.
This document has been prepared so that we can share success stories on our learners. In
addition, we will synchronise our working system so that we resonate from the same hymn book
and mould our own culture in setting of examination test items that meet the exact requirements
from the National Examining Body An attempt has been made to reflect on the good and poorly
designed test items. l hope you are going to enjoy reading the work and gain some important
information that will help you to do the right thing.
The most important factor in item writing is critical thinking.
What is critical thinking?
This is the process of analyzing and understanding how and why we reached certain conclusions.
In order to be a critical thinker, One must be able to:

•Recognize problems, to find workable means for meeting those problems.

•Understand the importance of prioritization and order of precedence in problem solving.
•Gather and marshal pertinent (relevant) information.
•Recognize unstated assumptions and values.
•Comprehend and use language with accuracy, clarity, and discernment.
•Interpret data to appraise evidence and evaluate arguments.

RICHARD PAUL QOUTE

The art of thinking about your thinking while you are thinking in order to make your thinking
better.

•More Clear
•More Accurate
•More Defensible
 Bloom’s taxonomy is key when we set lest items.

Bloom's Taxonomy Revised Terms

Remembering: Retrieving, recognizing, and recalling relevant knowledge from long-term
memory.

Understanding: Constructing meaning from oral, & written, and graphic messages
interpreting, exemplifying, classifying summarizing, inferring, comparing, and explaining

Applying: Carrying out or using a procedure through executing or implementing

Analyzing Breaking material into constituent parts, determining how the parts relate to
one another and 10 an overall structure or purpose through differentiating, organizing,
and attributing.

Evaluating: Making judgments based on criteria and standards although checking and
critiquing

Creating: Putting elements together to form a coherent functional whole; reorganizing

elements into a new pattern or structure through generating, planning, or producing.
(Anderson & 2001, 67-68)

However, in item design, the syllabus has objectives which have the key words that the learners
should observe so that they will meet the objective set and the teacher can safely monitor
performance lagging gaps and plug loopholes before final examination.
 •Remembering: Can the student recall or remember the information? define, duplicate, list,
memorize, recall, repeat, reproduce, state

•Understanding: Can the student explain ideas or concepts? classify, describe, discuss, explain,
identify, locate, recognize, report, select, translate, paraphrase

•Applying: Can the student use the information in a new way? choose, demonstrate, dramatize,
employ, illustrate, interpret, operate, schedule, sketch, solve, use. write.

•Analyzing: Can the student distinguish between the different parts? appraise, compare, contrast,
criticize, differentiate, discriminate, distinguish, examine, experiment, question, test

•Evaluating: Can the student justify a stand or decision? appraise, argue, defend, judge, select,
support, value, evaluate

•Creating: Can the student create new product or point of view? assemble, construct, create,
design, develop, formulate. write

TESTING COMMITTE
School Testing committee to design and revise the school's testing policies
•Examination Development Administration
•Determine acceptable statistical parameters for Test Items and Examinations.

WRITING STYLE PROTOCOL

Establishing clear guidelines for the school leaves little room for ambiguity; helps ensure
uniformity in the presentation Of examination throughout the curriculum: Ill case Of school set up
the examination committee takes responsibility as such

•Will present or past tense be used for test items?

•Will options end in periods, whether or not there are complete sentences?
•Will all options begin With a capital letter?
•When stressing a word in the stem, will it be highlighted, boldfaced, italicized, underlined?
•Will the term learner or student be used?

If a test is not reliable and valid, then the student scores are not reliable and valid and not indicative
of the students understanding of the topic

Reliability is the consistency Of the

measurement A test (or test item) is
reliable, when high-scoring test-takers
consistently answer the question(s)
correctly and low scoring test takers
consistently answer the questions
incorrectly. Reliability is determined
by the same students taking the same
exam today that they took yesterday. If
they make the same score on the
second day that they made on the first
day, the test is reliable Right?

•WRONG—they would have previous experience with the exam and such experience would
influence their scores on the second examination experience.
•Statistical measures that describe the internal consistency (reliability) of examination scores using
the result of only one test

Validity
Does the test measure what it claims to measure?

According to the Standards for Educational and Psychological Testing (1999)

"Validity . . . refers to the appropriateness, meaningfulness, and usefulness of the inferences made
from test scores." Therefore, it is not a test, but rather the inferences drawn from a test that are
ultimately evaluated for validity.
Validity

The presence of even a few poor items on an exam reduces reliability, sometimes to a great extent
Item Analysis

The more items you have, the greater the chance of poor test items.
D Length of examination
50 items

Actions for poor test items

I .Give credit for more than one answer.
Total number of items remains the same; many students' examination scores are likely to
increase.
2.Nullify test item by giving credit for all choices.
Total number of items remain the same; scores of students who answered it incorrectly will
increase and students who answered it correctly will stay the same.
3 .Delete the item from the examination.
IA Total number of items decreases, scores of student that answered it incorrectly will increase;
scores of students who answered it correctly will decrease.

ITEM WRITING RULES

Get rid of names
Get rid of 'multiple' multiples
Use non-sexist writing style
Develop parsimonious writing style
Cross out "of the following"
Delete scenarios
Write items independent of each other
Use a question format when possible
Make distracters plausible and homogeneous
Equal in length
No opposites
EIiminate "all of the above" and "none of the above'
Rewrite any "all except" questions
Ensure that alternatives do not overlap
Vary correct answer

NB. Now that you have read and understood the background behind making or setting the Item
Writing Guidelines
General Guidelines
l) Do Not Use Trick Items
• Items are tricky either intentionally or accidentally.
• Trick items are unfair to examinees and threaten the validity of the test.
• Trick items measure test-taking skills more than the intended construct.
• Trick items heighten test anxiety and cause examinees to mistrust the intent of all other items.
• Causes of unintentional trickiness.
• Trivial content o Discrimination between options is too fine o Overlapping options o Irrelevant
content o Single answer allowed, but multiple correct answers possible o Ambiguity in either
the stem or options
2) Measure a Single Construct
• If an examinee incorrectly answers an item that has multiple constructs, it is impossible to
know which construct is not mastered.
• Items are generally scored dichotomously, so the only inference that can be made is that the
examinee knows the entire item or none of the item.
• Compound items heighten test anxiety and can lower the perceived validity of the exam.
3) Avoid Opinion-Based Items
• Never ask "What would you ... do" use" try", etc. The examinee's answer can never be wrong.
• Use caution when asking for the "best" thing, or the "best" way of doing something, unless it is
clearly the best amongst the options.
• If differences exist between any experts' opinion about what the "best" is, then avoid using it.
• Qualify the standard for "best" (i.e., according to ... )
4) A void Absolute Modifiers such as always, never, only and none.
• The use of absolute modifiers in options makes it easy to eliminate options, increasing the
guessing probability.
5) Avoiding Excessive Verbiage
• "Verbosity is an enemy to clarity. " (Haladyna, 2004)
• Wordy items take longer to read and answer, meaning fewer items can be presented in a fixed
amount of time, reducing reliability.
• Write items as briefly as possible without compromising the construct and cognitive demand
required.
• Get to the point in the stem and present clean, clear options for the examinee to choose.
• Avoid unnecessary background information
6) Avoid over-specific or over-general content
• Over-specific content tends to be trivial
• Over-general content tends to be ambiguous.
7) Use Novel Content
• Do not repeat exact wording from training materials.
• Repeated wording tends to test recall and recognition, rather than learning.

10) Keep Items Independent

• Content should be independent from item to item.
• Don't give the answer away to one item in the stem of another.
• Don't make answering one item correctly dependent on knowing the correct answer to another
item.
I l) Write items to a sixth grade reading level
• Use appropriate vocabulary for construct
• Use necessary technical terms & content
• For everything else, use the simplest words and sentence structure possible
12) Do not teach.
• The purpose of certification is to verify knowledge.
• Do not introduce new material nor reinforce material the examinee should already know.
Guidelines for Writing the Stem
l) Write the stem in the form of a question.
2) Place the main idea in the stem.
3) The examinee should be able to know immediately what the focus of the item is by just reading
the stem.
4) The examinee should be able to answer the question without reading the options.
5) Make the stem a complete sentence, containing all information necessary to answer the
question.
6) Keep the stem as brief as possible.
7) Move repeated words from the options into the stem, when possible
8) Avoid negative words such as "not" or "except"
Guidelines for Writing the Options
l) Make the correct answer always correct.
2) Make the distractors always wrong, but attractive (plausible) to examinees that are not
minimally competent.
3) Avoid "All of the above" and "None of the above" as options.
4) Avoid inadvertent clues.
• Do not always make the correct option the longest.
• Do not repeat words in the options that are in the stem (clang associations).
• Don not use specific determiners, such as, always, never, all, every, etc. They are so extreme that
they are seldom the correct answer.
• Keep options homogeneous (parallel) in content, length, and grammatical structure.
5) All items should have at least one correct option and one distractor.
6) At least four options are preferable, but three are sufficient.
7) Order the options in either logical or numerical order, if one exists. Otherwise, sort shortest to
longest.
8) Avoid humour.
Guidelines for Writing Distractors
l) Use logical misunderstandings or misconceptions.
 2) Use common errors.
3) Use familiar terms, key-words, structures, or ordering.
4) Use statements that are correct or true but do not answer or address the stem (question) 5) Avoid
opposing statements.
6) Use correct concepts, but "mixed up." reliable, valid and credible test items according to the
curriculum syllabus, there is need to get a dummy of well constructed and poorly constructed
test item.
What leads to constructing poor test items
I Designing ambiguous test items that are difficult to comprehend.
2. Duplicating test items that leads to questions going outside the prescribed assessment
objectives.
3. Not following the syllabus objectives.
4. The teacher will not consider to follow the examination blue print and the blooms
taxonomy in setting the questions level of difficulty to cater for individual difference in a
class of learners.
5. Lack of thorough knowledge of the subject content of a grade zone.
6. Test item that are linked and give a clue or leads which affects validity
7. Failure to use the proper term that reflect the intended behavioural change on the part of
the learner e.g identify, simplify, list etc
8. Creating ambiguous pictures and diagrams without clear instructions.
9. Poor distribution of questions and marks on the covered are to be tested.
10. Content to be covered in test should start from ECD to the current grade level in order to
give teachers and learners a clear picture or reflection of the content covered from the
taught concepts in the previous grades.
General Guidelines.
a. Do not use trick items.
b. Measure a single construct.
c. Avoid opinion based items.
d. Avoid absolute modifiers such as always, never, only and none.
e. Avoid excessive Verbiage.
f. Avoid over specific or over general content.
Example or a well constructed Item, PE,
1. In time signature the top number shows the number of A. bars
on the stave. BE crotchet beats in a bar. C. rounds to be
repeated. D. pitch in a score.
(i) Syllabus Objective : Grade 4 page 27 — identify dotted notes, rests and simple time signature.
(ii) Syllabus Objective : Grade 5 page 42 — use simple compound time signatures in composing
rhythms.
 (iii) Syllabus Objective : Grade 6 page 56 — use simple and compound time signatures in
music. (iv) Syllabus Objective : Grade 7 page 69 — read rhythmic and melodic notation in
simple and compound time signatures.
Assessment Objective : page Sl — become literate in music and dance.

2. Ill which dance are Women the main performers?

A. amabhiza.
B. chinyambera.
C. isitshikitsha.
D. muchongoyo.
(i) Syllabus Objective : Grade 3 page 15 — identify dance songs and artefacts of other cultures
(ii) Syllabus Objective Grade 4 page 26- identify past and present dances from various cultures
in Zimbabwe
•Grade 4 page 26• relate dances to Zimbabwean geographical location.
(iv) Syllabus Objective ; Grade 5 page 40-o•utline dances performed by and
children,
(v) Syllabus Objective : Grade 6 page 55- identify different features of dance from various
cultures around Zimbabwe.
Assessment objective page 81 — identify and post-colonial era
and visual arts.

3. A person who teaches creative dance movements is

called A. an actor. B. an adjudicator. C. a choreographer.
D. a scriptwriter.
(i) Syllabus Objective . Grade 3 page 17' design dance sequences or choreography from
songs taught in
(ii) Syllabus Objective : Grade 4 page 29 — master the dance movements and patterns L
(iii) Syllabus Objective : Grade 5 page 44 — create complex sequences of movements with
greater focus *force and intent
Syllabus Objective Grade 6 page 58-demonstrate the ability to movements With
different musical rhythms and styles
Syllabus Objective ; Grade 7 page SO — identify careers ill dance.
Assessment Objective : page 81 — design dance sequences(choreography) from songs taught in
class.

Mathematics test item.

l . Which one represent 34,68 in words.
A. Thirty four comma sixty-eight
B, Three four Six eight
C Thirty Four comma six eight
D.Thirty six comma four eight
(i) Syllabus Objective: Grade 3 page 7: Read and write any number in numerals and words in
the range:
(ii) Syllabus Objective: Grade 4: page 17: identify, read and write any number in numerals and
words in the range.
(iii)Syllabus Objective: Grade 5: Page 27 identify, read and write any number in numerals and
words in the range
(iv) Syllabus Objective: Grade 6: Page 38 identify, read and write any number in numerals and
words in the range
(v) Syllabus Objectives: Grade 7: Page 49 identify, read and write any number in numerals and
words in the range
Assessment Objective : page 61 — recall, recognise and use mathematical symbols, terms and
definitions.

Example of a poorly constructed test item.

l. Which country started the Roman Numeral system? A

Zimbabwe
Italy
C. Europe
D. Rome
The test item is not testing any item related to any objective or the assessment objective
from the syllabus and is rendered a wrong test item,
2. Five ten is the same as

B 50
c. 0,50

NB: This test item has not test a clear objective it intends to test. lt can be interpreted
as five sets of 10 which brings the answer 50, or it can be reduced to lowest terms and
a half is regarded the answer. Moreso, the same half is represented as 0,5.The other
learner may put Roman numeral as he might have grasped concept on Roman
Numerals. ln a nutshell the test item is tricky, or vague. lt is not a good thing to develop
unclear test items.
The above has just been a sample and more examples shall be displayed during
the training development session.

The document has outlined the strength and weakness that we seek to highlight so that
we can do the right thing setting of papers which are fool proof and item to measure an
objective dimension it purports to measure and bring about good results that are
credible, reliable and valid.

Mr. M. Madanhire Real Mist Educational Consultancy. Copyright © 2024