M402U3=Test Development Process for teaching

TEST DEVELOPMENT PROCESS
Facilitated by:
Mr. Chan Horm
Chapter 2:
Chapter 2: UNIT 3
UNIT 3
1 TM402 Unit3
Test Development Process

Stages of Test Development Process
• Different Test Experts might suggest different stages
or processes. Thus, the following stages are
suggested by Shohamy and Glatthorn.
2 TM402 Unit3
1.Assess the constraints and resources
2. Determine the purpose of the test
3. Determine the objective of the test

3 TM402 Unit3
5. Determine the relative weights of all
objectives and different parts of the test
6. Determine the test method(s) and procedures
6. Determine the test method(s) and procedures
4. Define the content areas of the test
7. Write test items and tasks assemble them,
7. Write test items and tasks assemble them,
and write directions
and write directions
8. Review and Revise the test
9. Develop scoring methods

Test Developers should make sure:
• Materials resources such as paper, newspaper,
photocopying machine(s) are available.
• Should have enough time to construct the paper; for
example, MCQs are not only difficult, but time-
consuming to construct.
• Financial resources are very important to support the
process.
• Human resources are the most important factors. Thus,
Test Developers should have been well-trained
4 TM402 Unit 3

• Tests should reflect the vital purposes of the Ss’ learning and Ts’
teaching.
• The different types of tests will serve different purposes of the test.
The information or results obtained from the tests will help
teachers and/or decision makers determine/ or:
- if students need special help
- if students are allowed to continue their studies
- check if students have mastered the materials
- measure language proficiency regardless previous training
courses
- discover how far students have achieved the course objectives.
- diagnose and/or assist students
5 TM402 Unit 3

The test objectives should be congruent with the course or
teaching objectives, but they won’t be covered everything.
1. to determine what have been taught.
2. to select the priority objectives.
Example: Objective: To be able to understand the reading
passage.
Purposes: - to understand specific information
- to identify the main ideas
- to extract specific details
- to understand specific words from the context.
Based on the test content, the objectives of the tests can be
specified.
6 TM402 Unit 3

4. Define the content areas of the test
• Defining the content areas of the test is based on the types of
tests. For example, for the progress test, the content items should
be those which have just been taught, while for final achievement
test, the content items can be selected in terms of their
importance. These can be called “syllabus-content approach
syllabus-content approach”.
• Thus, it is impossible to test all
impossible to test all of what has been taught.
• In short, the test items
the test items should be based on the objectives of the
course or determine the test objectives
determine the test objectives and then specify the
content items which are included in the test.
• Determine an appropriate balance between linguistic content and
skills.
7 TM402 Unit 3

• Generally, the content of test is based directly on a
course syllabus or materials used during instruction.
In this sense, the test material should be similar to
what has been taught( Shohamy, 1985)

• Before test items can be written, it is important to decide:
-lengths and weights of objectives
-different content items and skills.
• The four macro-skills should be given different emphases and have
different weightings at different times. Example:
-In classroom testing- weights assigned depend entirely on the
teacher. E.g. T spends a lot of time on a certain aspect, but she
tests a certain priorities.
-In external tests-Decision makers set the criteria of weights based
on the importance given to the aspects in the curriculum rather
than regarding the previous learning experiences. (Study a Table of
Specification)
9 TM402 Unit 3

A Table of Specification
Objectives
Content
Using language
accurately
Using lexical items
correctly in context
Communicating the
message
The entire letter 50% 20% 30%

Objective
Content
Using language
accurately
Using lexical items
correctly in context
Communicating
the message
Using the
appropriate
register
The entire
letter
15% 20% 50% 15%

1. The table of Teacher number 1 reflects a
structural approach to teaching. The main
emphasis is placed on language accuracy(50%).
Socio-linguistic aspects are discarded completely.
2. The table of Teacher number 2 reflects
communicative approach to teaching. The main
emphasis is placed on communicating the
message(50%) and only 15% of the total score is
assigned to language accuracy.

6. Determine the test method(s) and
6. Determine the test method(s) and
procedures
procedures
• What assignment is used to get information from students?
- proficiency test (mixture of direct and indirect, discrete-point
and integrative testing).
- achievement tests(syllabus content approach)
- placement test (discrete-point testing)
• In this phase, one needs to determine a certain approaches:
=>the objectives of the test
=>types of test tasks (discrete point or integrative test)
• The test procedures can be determined after the types of test
are decided.
• Test constructors have to know the different types of test and
testing as well as the types of test items.
13 TM402 Unit 3

7. Write test items and tasks assemble
7. Write test items and tasks assemble
them, and write directions
them, and write directions
• It is important to decide which items and task types should be
included
=>The types of test and tasks affect the test takers’ level of
performance and their competence.
=> Keep balance the objectives of objective testing and subjective
testing in order to achieve the suitable degree of both reliability
and validity.
14 TM402 Unit 3

• When the test has been constructed, its items need to be
reviewed and revised in order to:
=>ensure the test validity, reliability, and authenticity
=>have positive impact on teaching and the examinees
• To achieve these individuals mentioned above, the review and
revision should involve in:
=> a number of teachers if possible those who were not engaged
in constructing the test in the first stage.
=> the checklist questions on page 72
page 72.
15 TM402 Unit 3

• Scoring method should be developed in order to achieve reliability.
=>Objective tests
Objective tests tend to produce more reliable results than
subjective ones.
=>Scoring methods
Scoring methods of subjective tests (e.g.: speaking and writing
tests) appear to be more difficult to achieve reliability.
• Developing the marking criteria of subjective tests tend to help
scorers increase inter-rater and intra-rater reliability.
• Intra-rater reliability
Intra-rater reliability refers to the degree to which a teacher or
examiner making subjective ratings of ability gives the same
evaluation on two or more different occasions.
• Inter-rater reliability
Inter-rater reliability refers to the degree to which different
examiners or judges making subjective ratings of ability agrees in
their evaluations of that ability.
16 TM402 Unit 3

Types of test Items
• Multiple Choice
• Paired Statements
• True/False Items
• Short-Answer Items
• Matching
• Cloze Procedures
• The C-Tests
• Transformation and Paraphrases
• Sentence Reordering
17 TM402 Unit3

 Dictation
 Transformation and Paraphrases
 Sentence Reordering
 Essay
 Prepared essay
 Open-book exams

MULTIPLE CHOICE QUESTIONS
MCQs have been considered ideal test instruments for
measuring students’ knowledge of grammar
&vocabulary, mainly because they are easy to score
easy to score
and, with the use of computer technology, the answers
can be read by machines, not people, with the
consequent elimination of scorer error
elimination of scorer error (thus, with
increased rater reliability).
19 TM402 Unit3

Features of MCQs:
Number of alternatives
The ideal # of alternatives is 5. A larger #, e.g. 7,would obviously
reduce the chance element, but it would be extremely difficult if
not impossible to construct as many as seven good options for
each item. Most classroom tests use 4 options, not 5, precisely
because of this difficulty.
Areas to be measured & number of items to be included
- Before starting the construction of any given test, the test-
developer must decide on a) the areas that the test is going to
measure with MCQs and b) the number of items that are going
to be included in the test.
20 TM402 Unit3

- The MCQ test must be long enough to provide evidence of the
test’s performance & short enough to be practicable.
• An excessively long test is undesirable because, apart from being
more difficult to administer, it would cause mental strain &
tension among tests.
• Generally, the # of items will depend on the level of difficulty,
the nature of the areas being tested, and the purpose of the test.
Usually, the teacher’s experience will determine the length of a
teacher’s experience will determine the length of a
test for
test for classroom use.
21 TM402 Unit3

Components of MCQs
Components of MCQs
1. The stem
1. The stem
The journalist was _____ by enemy fire as he tried to send a story by
The journalist was _____ by enemy fire as he tried to send a story by
radio.
radio.
2. The responses (also alternatives/options)
a). wronged
a). wronged b). wounded
b). wounded c). injured
c). injured d). damaged
d). damaged
One option is the answer/correct option/key
answer/correct option/key, while the others are
called distractors
distractors, because their function is
, because their function is to distract most poor
students (i.e. those who do not know the answer) away from the
correct option.
22 TM402 Unit3

Principles of MCQ test construction
Principles of MCQ test construction
• Each MCQ item should have only one correct answer. This answer
must be ABSOLUTELY CORRECT unless the instruction says “choose
the best option.”
• Normally, nobody tests grammar & vocabulary at the same time,
but sometimes word order and sequence of tenses are tested
simultaneously. These are impure MCQ
impure MCQ items:
23 TM402 Unit3
I never knew where _____.
a). had the boys gone b). the boys have gone
c). have the boys gone d). the boys had gone

MCQs Options
• Each option should be grammatically correct
grammatically correct when placed in
the stem, except of course in the case of specific grammar
test items. What’s the problem with the following item?
This can be corrected in at least two ways? How?
• Stems
Stems ending in preps can also present some difficulties. In
the following reading comprehension item
reading comprehension item, one option can be
immediately ruled out. Which one?
24 TM402 Unit 3
John soon returned to _____.
a). work b). the prison c). home d). school
Someone who designs houses is a(n) _____.
a). designer b). builder c). architect d). Plumber

MCQs Forms
Forms:
Forms:
a) an incomplete statement/the single completion
a) an incomplete statement/the single completion
b) a complete statement
b) a complete statement
c) a question
c) a question
25 TM402 Unit 3
He accused me of _____ lies.
a). speaking b). saying c). telling d). talking
Everything we wanted was to hand.
a). under control b). within reach
c). well cared for d). being prepared for
According to the writer, what did Tom immediately do?
a). He ran home. b). He met Bob.
c). He began to shout. d). He phoned the police

MCQs Contents
Contents:
Contents: A MCQ should contain those words or phrases which
would otherwise have to be repeated in each option.
The stem should be rewritten so that it reads:
26 TM402 Unit3
The word ‘astronauts’ is used in the passage to refer to
a). travelers in an ocean liner. b). travelers in a space-ship.
c). travelers in a submarine. d). travelers in a balloon
The word ‘astronauts’ is used in the passage to refer to travelers in
a). an ocean liner. b). a space-ship.
c). a submarine. d). a balloon.

MCQs Designs
The same principle applies to grammar items.
The item should be rewritten like this:
The first of these two items would be correct only if one of the
errors made by the students in their free written work is the
omission of the preposition after look (a common error), i.e. if it’s
clear they don’t know that a preposition is necessary after this
verb.
27 TM402 Unit 3
I enjoy _____ the children playing in the park.
a). looking to b). looking about
c). looking at d). looking on
I enjoy looking _____ the children playing in the park.
a). to b). about c). at d). on

MCQs Key
THE CORRECT ANSWER
THE CORRECT ANSWER
• There should be no doubt as to the correct answer. Thus, each item
each item
should be checked by another person
should be checked by another person.
• The correct answer should be of approximately the same length
approximately the same length
as the distractors, especially in
as the distractors, especially in vocabulary tests & tests of reading
& listening comprehension. So, avoid tendency to make correct
avoid tendency to make correct
answer longer than distractors simply
answer longer than distractors simply because it’s necessary to
qualify a statement or word to make it absolutely correct. Here’s an
example of such ‘giveaway’ item:
28 TM402 Unit 3
He began to choke while he was eating the fish.
a). die b). cough and vomit c). grow very angry
d). be unable to breathe because of something in the windpipe

Advantages Disadvantages
-less time to answer - difficult to construct
-easy to mark - difficult to come up with
-objectively marking alternative responses
-high reliability - only a simple act of recog-
-versatile at all levels nition and recall
- cheating may be
facilitated

Paired Statements
Test takers are asked to indicate the correct relationship
between the pairs of statements. Example: page 57
Suggested Answers:
30 TM402 Unit 3
True/False Items
True/False Items
1 2 3 4 5 6 7
B A C C A B B
• Test takers are asked to select one of two given responses.
• It is difficult to distinguished in Ss’ response, between unfortunate guessing
and misinterpretation of the questions.
• There are several variants:
• T/F
T/F T/F or TF
T/F or TF
• Draw a circle around TF if the statement is true under some
Draw a circle around TF if the statement is true under some
circumstances but false in others.
circumstances but false in others.

Advantages Disadvantages
-can test large amounts -easy
of contents -difficult to discriminate
-less time to answer students that know the
material and students
who don’t
-50-50 chance of getting
the right answer by
guessing

Short-Answer Items
The question is to be given a short answer. Thus, the long one will be
rejected.
Advantages
-easy to construct
-good for "who, ” “what,” ”where, "content”
-minimize guessing
-encourage more intensive study
Disadvantages
-may overemphasize memorization of facts
-questions may have more than one correct answer
-scoring is laborious
32 TM402 Unit 3

Matching
Matching
•items consist of a column of Premises and a
items consist of a column of Premises and a
column of responses.
column of responses.
•Directions for matching the two column have to
Directions for matching the two column have to
be unambiguous.
be unambiguous.
•Students need to read the stem or premise and
Students need to read the stem or premise and
scan the options for the correct answer.
scan the options for the correct answer.
•It is easy to construct.
It is easy to construct.

Types of Matching Items
-terms with definitions
-phrases with other phrases
-causes with effects
-parts with larger units
-problems with solutions

Advantages
-Maximum coverage at knowledge level
-Valuable in content areas that have a lot of
facts
Disadvantages
-Time consuming for students
-Not good for higher levels of learning

Example: Match the terms on the left with
their definitions on the right.
1.Proficiency tests a. …………
2.Achievement tests b. ………..
3.Placement tests c. …………
4.Diagnostic test d. …………

37 TM402 Unit 3
Questions and Answers
Thank you very much!!!

Cloze tests
• In its purest form, a cloze consists of the deletion of every nth
deletion of every nth
word in a text
word in a text (somewhere but every fifth
fifth or tenth
tenth word).
Because the procedure is random
the procedure is random, it avoids test designer failings
avoids test designer failings.
Example of cloze fragment:
• Modified cloze is useful for placement tests, achievement or
proficiency tests.
38 TM402 Unit 3
They sat on a bench attached 1 _____ a picnic table. Below
them they 2 _____ see the river gurgling between overgrown 3
_____. The sky was diamond blue, with 4 _____ white clouds
dancing in the freshening 5 _____. They could hear the call of 6
_____ and the buzzing of countless insects. 7_____ were
completely alone.

THE CLOZE PROCEDURE
• Because of the randomness of the deleted words, anything may be
anything may be
tested within a singles cloze text: grammar
tested within a singles cloze text: grammar, collocation, fixed
phrases, reading comprehension…which makes it, at least on the
face of it, the perfect testing instrument.
PROBLEMS WITH THE CLOZE-PROCEDURE
PROBLEMS WITH THE CLOZE-PROCEDURE
However, …
- the score obtained by the student depends on
score obtained by the student depends on the particular words
words
that have been deleted, rather than on
deleted, rather than on their general knowledge of
knowledge of
the L2
the L2
- some items
some items are more difficult
more difficult to supply than others
than others
- in some cases, there may be several possible answers
several possible answers
39 TM402 Unit 3

THE CLOZE PROCEDURE
• In spite of these reliability problems, supplying the correct word
supplying the correct word
for a blank does imply
for a blank does imply …
- an understanding of context
understanding of context &
- a knowledge of that word and how it functions
knowledge of that word and how it functions, which makes
the cloze technique a very useful technique to use in language
tests.
CLOZE OR FILL-IN-THE-BLANKS?
CLOZE OR FILL-IN-THE-BLANKS?
Cloze tests look similar to completion or blank-filling tests, but
they are different. In blank-filling tests
blank-filling tests, the words for deletion
the words for deletion
are selected subjectively
are selected subjectively (consisting largely of structural words
in grammar tests & key content words in vocab. or reading tests).
In cloze tests
cloze tests, however, the words
the words are deleted systematically
deleted systematically.
40 TM402 Unit 3

CLOZE CONSTRUCTION
• Once the text has been chosen, the construction of the cloze is
purely mechanical
purely mechanical:
- every “nth
nth” word is deleted;
- deletion interval: commonly but every 5th
5th & every 10th
10th word.
BUT, if every 7th
7th word has been deleted in the first few
sentences, that is the interval that should be used for the rest of
the text.
- 5th
5th, 6th
6th, and 7th
7th word intervals are the preferred, mainly
because a shorter interval would make it very hard for the
student to just understand the text, since there would not be
enough context.
41 TM402 Unit 3

CLOZE CONSTRUCTION
• If, on the other hand, every 10th
10th or 12th
12th word is deleted, it would
be necessary to have a very long text.
E.g.1: 40 deletions every 10th
10th word =>280
280 to 300
300-word text.
E.g.2: 40 deletions every 12th
12th word => 480
480 to 500
500-word text.
42 TM402 Unit 3

LEVEL OF TEXT DIFFICULTY
The level of difficulty of the text is very important: if the text is
already difficult to read without blanks, imagine how difficult it
would be once the blanks are inserted!
The difficulty level
difficulty level of the text is affected by as many as the
the
following variables
following variables:
-text length;
-amount of time allowed to complete the task;
-learner familiarity with vocabulary & syntax of the passage;
-length & complexity of the sentences in the passage;
-learner familiarity with topic & with discourse genre of text
(content & formal schemata
content & formal schemata)
-blank interval (every 5th
5th word versus every 9th
9th word, for
instance)
43 TM402 Unit 3

THE C-TEST
The C-Test is really a variety of cloze. In stead of deleting the
whole words, it is the second half of every second word which is
deleted. (Example in your textbook, page 65)
The following exercise is example of deletion methods.
44 TM402 Unit 3
This is a variation on the cloze test, in which the students read a
brief paragraph in the L2. The first two sentences are left intact.
There_ _ _ _ _, every ot_ _ _ word i_ printed int_ _ _, but f_ _ each
alte_ _ _ _ _ word, on_ _ the fi_ _ _ half o_ the wo_ _ is wri_ _ _,
and t_ _ second ha_ _ is indi_ _ _ _ _ by a bl_ _ _ space repres_ _ _
_ _ _ each let_ _ _. T_ _ students’ abi_ _ _ _ to fi_ _ in t_ _ blank
spa_ _ is tho_ _ _ _ to b_ a mea_ _ _ _ of th_ _ _ language profi_ _
_ _ _ _.

A Variety of Cloze
Full Text of the C-Test
45 TM402 Unit 3
This is a variation on the cloze test, in which the
students read a brief paragraph in the L2. The first two
sentences are left intact. Thereafter, every other word
is printed intact, but for each alternate word, only the
first half of the word is written, and the second half is
indicated by a blank space representing each letter.
The students’ ability to fill in the blank space is thought
to be a measure of their language proficiency.

Transformation & Paraphrases
Transformation
Transformation = rewriting sentences in a slight different form, retaining meaning
rewriting sentences in a slight different form, retaining meaning
of original sentence. The next item tests knowledge of verb & clause patterns
triggered by the use of I wish:
46 TM402 Unit 3
Finish each of the following sentences in such a way that it is as similar as
possible in meaning to the sentence printed before it.
I’m sorry that I didn’t get her an anniversary present.
I wish ___________________________.
To complete the task successfully, the student has to
To complete the task successfully, the student has to
understand the original sentence & know how to construct a
understand the original sentence & know how to construct a
grammatically correct equivalent. Thus, this type of test item
grammatically correct equivalent. Thus, this type of test item
informs teachers about student knowledge of the
informs teachers about student knowledge of the
language system.
language system.

SENTNECE REORDERING
• Getting the students to put words in the right order to make
appropriate sentences informs the teacher about their knowledge
knowledge
of syntax and lexico-grammatical elements and mechanisms
of syntax and lexico-grammatical elements and mechanisms.
Advantage
Advantage: Reordering items are fairly easy to construct.
Problem:
Problem: It’s difficult to ensure only one correct order.
47 TM402 Unit 3
Put the words in order to make correct sentences.
called / I / I’m / in / sorry / wasn’t / when / you
______________________________________
I wasn’t in when you called I’m sorry

The usefulness of a test can be judged
according to six qualities.
a) reliability d) impact
b) construct validity e) practicality
c) authenticity f) interaction

Reliability
Reliability ( In testing) a measure of the degree to
which a test gives consistent results. A test is said to
be reliable if it gives the same results when it is
given on different occasions or when it is used by
different people.
the consistency of test results in different testing
situations

Construct validity
Construct validity: the extend to which we can
interpret a given test score as an indicator of the
ability (ies), or construct(s), we want to measure.
A test measuring what it is intended to measure
 Validity: The extent to which inferences made from
a test are appropriate, meaningful, or useful

Authenticity
Authenticity: the degree of correspondence
between the characteristics of a given language
task to the features of a target language use task.
Stimulating real-life situations

Impact
Impact: the effect of a test on individual test taker,
other stakeholder, (e.g. teacher, parent, school
administrator, or test developer).
Washback direct effect of testing on teaching
and individuals

Practicality
Practicality: the practical issues related to testing,
time, resource, administrative logistics. One needs
to consider the relationship between the resources
required for the design, construction, and
administration of the test and the resources
available for these activities.
Resources required for the design, construction,
and administration of the test.

54 TM402 Unit 3
Questions and Answers
Thank you very much!!!

M402U3=Test Development Process for teaching

Recommended

More Related Content

Similar to M402U3=Test Development Process for teaching (20)

Recently uploaded (20)

M402U3=Test Development Process for teaching