Testing Principles

By
Didi Sukyadi
English Education Department
Indonesia University of Education
 Practicality
• Is not excessively expensive
• Stays within appropriate time constraints
• Is relatively easy to administer
• Has a scoring/evaluation procedure that is
specific and time efficient
• items can be replicated in terms of resources
needed e.g. time, materials, people
• can be administered
• can be graded
• results can be interpreted
 Reliability
• A reliable test is consistent and dependable.
• Related to accuracy, dependability and
consistency e.g. 20°C here today, 20°C in North
Italy – are they the same?
According to Henning [1987], reliability is
• a measure of accuracy, consistency,
dependability, or fairness of scores resulting from
the administration of a particular examination
e.g. 75% on a test today, 83% tomorrow –
problem with reliability.
 Reliability
• Student Related reliability: the deviation of an
observed score from one’s true score because of
temporary ilness, fatigue, anxiety, bad day, etc.
• Rater reliability: two or more scores yield an
inconsistent scores of the same test because of lack
attention on scoring criteria, inexperience, inattention,
or preconceived bias.
• Administration reliability: unreliable results because of
testing environment such as noise, poor quality of
cassettee tape, etc.
• Test reliability: measurement errors because the test is
too long.
 To Make Test More Reliable
• Take enough sample of behaviour
• Exclude items which do not discriminate well
between weaker and stronger students
• Do not allow candidate too much freedom.
• Provide clear and explicit instructions
• Make sure that the tests were perfectly laid out
and legible
• Make candidates familiar with format and testing
techniques
 To Make Test More Reliable
• Provide uniform and undistracted conditions
of administration
• Use items that pemit objective scoring
• Provide a detailed scoring key
• Train scorers
• Identify candidate by number, not by name
• Employ multiple, independent scoring
 Measuring Reliability
• Test retest reliability: administer whatever the test
involved two times.
• Equivalent –forms reliability/parallel-forms
reliability: administering two different bu equal tests
to a single group of students (e.g. Form A and B)
• Internal consistency reliability: estimate the
consistency of a test using only information internal
to a test, available in one administration of a single
test. This procedure is called Split-half method.
 Validity
• Criterion related validity: the degree to which results
on the test agree with those provided by some
independent and highly dependable assessment of
the candidates’ ability.
• Construct validity: any theory, hypothesis, or model
that attempts to explain observed phenomena in
our universe and perception; Proficiency and
communicative competence are linguistic
constructs; self-esteem and motivation are
psychological constructs.
 Reliability Coefficient
• Validity coefficient to compare the reliability of different
tests.
• Lado: vocabulary, structure, reading (0,9-0,99), auditory
comprehension (0,80-0,89), oral production (0,70-0,79)
• Standard error: how far an individual test taker’s actual
score is likely to diverge from their true score
• Classical analysis: gives us a single estimatefor all test
takers
• Item Response theory: gives estimate for each
individual, basing this estimate on that individual’s
performance
 Validity
• The extent to which the inferences made from
assessment results are appropriate, meaningful and
useful in terms of the purpose of the assessment.
• Content validity: requires the test taker to perform
the behaviour that is being measured.
• Content validity: Its content constitutes a
representative sample of the language skills,
structures, etc. With which it is meant to be
measured
 Validity
• Consequential validity: accuracy in measuring
intended criteria, its impacts on the
preparation of test takers, its effects on the
learner, and social consequences of test
interpretation and use.
• Face validity: the degree to which the test looks
right and appears to the knowledge and ability
it claims to measure based on the subjective
judgement of examinees who take it and the
administrative personnel who decide on its use
and other psychometrical observers.
 Validity
Response validity [internal]
• the extent to which test takers respond in the way
expected by the test developers

Concurrent validity [external]

• the extent to which test takers' scores on one test relate to
those on another externally recognised test or measure
Predictive validity [external]
• the extent to which scores on test Y predict test takers'
ability to do X e.g. IELTS + success in academic studies at
university
 Validity
• 'Validity is not a characteristic of a test, but a
feature of the inferences made on the basis of test
scores and the uses to which a test is put.'
• To make test more valid:
1) Write explicit test specification
2) Use direct testing
3) Scoring of responses related directly to what is
being tested.
4) Make the test reliable.
 Washback
• The quality of the relationship between a test
and associated teaching.
• We have positive effect and negative effect.
• Test is valid when it has a good washback
• Students have ready access to discuss the
feedback and evaluation you have given.
 Washback
• The effect of testing on teaching and learning
• The effect of test on instruction in terms of how
students prepare for the test
• Formative test: provides washback in the form of
information to the learner on progress toward goals,
while Summative test is always the beginning of
further pursuits, more learning, more goals
• To improve washback: use direct testing, use criterion
reference-testing, base achievement tests on
objectives, and make sure that the tests are
understood by students and teachers.
 Evaluation of Classroom Tests
• Are the test procedures practical?
• Is the test reliable?
• Does the procedure demonstrate content
validity?
• Is the procedure face valid and biased for
best?
• Are the test tasks as authentic as possible?
• Does the test give beneficial washback?
 NRT and CRT
• Is designed to measure the global language abilities such as
overall English Proficiency, academic listening ability, reading
comprehension, and so on.
• Each student’s score on such a test is interpreted relative to
the scores of all other students who took the test with
reference to normal distribution
• Criterion reference test is usually produced to measure well-
defined and failrly specific instructional objectives
• The interpretation of CRT is considered as absolute in a
sense that each student’s score is meaningful without
reference to the other students’ scores
 NRT and CRT
Characteristics NRT CRT
Types of interpretation Relative Absolute
Type of measurement To measure general To measure specific
language abilities objective-based language
points
Purpose of testing Spread students out a long Assess the amount of
a continuum of general material known or learned
abilities of proficiencies by each student
Distribution of scores Normal distributiom Varies; often non normal.
Test structure A few relatively long A series of short-well
subtest with a variety of defined subtests with
item content similar item contents
Knowledge of questions Students have little or no Student know exactly what
idea of what content to content to expect in test
expect in test items items
 Test and Decision Purposes
TYPES OF DECISION
NORM-REFERENCED CRITERION-REFERENCED
Test Qualities Proficiency Placement Achievement Diagnostic
Detail of Very general general specific Very specific
information
Focus General skills From all levels & Terminal Terminal and
prerequisite to skills of program objectives of enabling
entry course objective
Purpose of To compare To find each To determine the To inform
Decision individual and student’s degree of students and
individual appropriate level learning for teachers of
advancement or weaker
graduation objectives
Relationship to Comparisons Comparison Directly related Related to
Program with other within program to objectives objectives need
institutions more worls
Interpretation Before entry and Beginning of End of courses Beginning and/or
When at exit program middle of courses
administered
score Spread of wide Spread of Overall number Percentage of
range of scores narrower, and percentage each objective in
program specific of objectives terms of
 Characteristics of communicative tests
• Communicative test setting requirements:
1) Meaningful communication
2) Authentic situation
3) Unpredictable language input
4) Creative language output
5) All language skills
• Bases for ratings
1) Success in getting meaning across
2) Use focus rather than usage
3) New components to be rated
Components of Communicative competence

• Grammatical competence (phonology, orthography,

vocabulary, word formation, sentence formation)
• Sociolinguistic competence (social meanings,
grammatical forms in different sociolinguistic contexts)
• Discourse competence (cohesion in different genres,
cohesion in different genres)
• Strategic competence (grammatical difficulties,
sociolinguistic difficulties, discourse difficulties,
performance factors)
 Discrete-point/Integrative Issue
• Discrete point: measures the small bits and
pieces of a language as in a multiple choice
test made up of questions constructed to
measure students’ knowledge of different
structure
• Integrative test: measures several skills at one
time such as dictation
 Practical Issues
• Fairness issue: a test treats every student the
same.
• The cost issue
• Ease of test construction
• Ease of test administration
• Ease of test scoring
• Interactions of theoretical issues
 General Guidelines for Item Formats
• correctly matched to the purpose and content of the item
• only one correct answer?
• written at the students’ level of proficiency
• Avoiding ambiguous terms and statements
• Avoiding negarives and double negatives
• Avoid giving clues that could be used in answering other items
• All parts of the item on the same page
• Only relevant information presented
• Avoiding bias of race, gender and nationality
• Let another person look over the item
 More than one correct answer
• The apple is located on or around
• A) a table C) the table
• B) an table D) table
- Two correct answers (A and C), wordy
(somewhere around), repeat the word table
inefficiently
 Multiple Choice
• Do you see the chair and table? The apple is
on _____ table.
a) A c) the
b) An d) (no article)

Option d (no article) will be easily detected as a

wrong option so it is not a good distracter.
 True-False
• According to the passage,
antidisestablismentarianism diverges
fundamentally from the conventional
proceedings and traditions of the Church of
England
* Containing too difficult vocabulary.
 Ambiguous Word
• Why are statistical studies inaccessible to
language teachers in Brazil according to the
reading passage?
• Accessible: language teachers get very little
training in mathematics and/or such teachers
are averse to numbers
• Accessible: the libraries may be far away.
 Double negatives
• One theory that is not unassociated with Noam
Chomsky is:
• A. Transformational generative grammar
• B. Case grammar
• C. Non-universal phonology
• D. Acoustic phonology
- Use one negative only
- Emphasize it by underline, upper case, or bold-
face. For example: not, NEVER, inconsistent
 Receptive response items
• True-False
1) the statement worded carefully enough so it can be judged without
ambiguity
2) absoluteness clues are avoided
• Multiple Choice
1) Unintentional clues are avoided
2) The distracters are plausible
3) Needless redundancy in the options is avoided
4) Ordering of the option is carefully considered
5) The correct answers are randomly assigned
• Matching
1) More options than premises
2) Options shorter than premises to reduce reading
3) Option and premise lists r elated to one central theme
 True-False
• Items should be worded carefully enough so it
can be judged without ambiguity
• Avoid absoluteness
• This book is always crystal clear in all its
explanation: T F
- allow the students to answer correctly without
knowing the correct response.
- Absolute clues: all, always, absolutely, never,
rarely, most often
 Multiple Choice
• Avoid unintentional clues
• The fruit that Adam ate in the Bible was an
____
A. Pear C. Apple
B. Banana D. Papaya
Unintentional clues: grammatical,
phonological, morphological, etc.
 Multiple Choice
Are all distracters plausible?
Adam ate _______
A. An apple C. an apricot
B. A banana D. a tire
 Multiple Choice
• Avoid needless redundancy
• The boy on his way to the store, walking down
the street, when he stepped on a piece of cold
wet ice and
A. fell flat on his face
B. fall flat on his face
C. felled flat on his face
D. falled flat on his face
 Multiple Choice
• More effective:
The boy stepped on a piece of ice and ______
flat on his face.
A. fell
B. fall
C. felled
D. falled
 Multiple Choice
• Correct answers should be randomly assigned
• Distracters like “none of the above”, “A and B
only”, “all of the above should be avoided
 Matching
• Present the students with two columns of
information; the students then must find and
identify matches between the two sets of
information.
• The information on the left-hand column is
called matching-item premise
• On the right hand column is called option
 Matching
• More options should be supplied than premises
so the students can narrow down the choices as
they progress through the test simply by
keeping track of the options they have used.
• Options should be shorter than premises
because most students will read a premise then
search through the options
• The options and premises should relate to one
central theme that is obvious to students
 Fill in Items
• The required response should be concise
• Bad item:
• John walked down the street ________
(slowly, quickly, angrily, carefully, etc.)
• Good item:
• John stepped onto the ice and immediately
____ down hard (fell)
 Fill in Items
• There should be a sufficient context to convey
the intent of the question to the students.
• The blanks should be standard in length
• The main body of the question should precede
the blank
• Develop a list of acceptable responses
 Short Response
• Items that the students can answer in a few
phrases or sentences.
• The item should be formatted that only one
relatively concices answer is possible.
• The item is framed as a clear and direct item
• E.g. According to the reading passage, what
are the three steps in doing research?
 Task Items
• Task item is any of a group of fairly-open ended item types
that require students to perform a task in the language that
is being tested.
• The task should be clearly defined
• The task should be sufficiently narrow for the time available.
• A scoring procedure should be worked out in advance in
regard to the approach that will be used.
• A scoring procedure should be worked out in advance in
regard to the categories of language that will be rated.
• The scoring procedure should be clearly defined in terms of
what each scores within each category means.
• The scoring should be anonymous
 Analytic Score for Rating Composition Tasks
20-18 17-15 14-12 11- 5-1
Excellent Good to Adequate to Unacceptable Not-
to Good Adequate Fair college
level work

Organization
(introduction,
body,
conclusion)

Logical
development
of ideas

Grammar
Punctuation,
Spelling,
mechanics

Style and
quality of
Holistic Version of the Scale for Rating
Composition Tasks
• Content
• Organization
• Language Use
• Vocabulary
• Mechanics
 Personal Response Items
• The response allows the students to
communicate in ways and about things that
are interesting to them personally
• Personal Responses include: self assessment,
conferences, porfolio
 Self-Assessment
• Decide on a scoring type
• Decide what aspect of students’ language
performance they will be assessing
• Develop a written rating for the learners
• The rating scale should decide concrete language
and behaviours in simple terms
• Plan the logistics of how the students will assess
themselves
• The students should the self-scoring procedures
• Have another student/teacher do the same scoring
 Conferences
• Introduce and explain conferences to the students
• Give the students the sense that they are in control
of the conference
• Focus the discussion on the students’ views
concerning the learning process
• Work with the students concerning self-image issue
• Elicit performances on specific skills that need to be
reviewed.
• The conferences should be scheduled regularly
 Portfolios
• Explain the portfolios to the students
• Decide who will take responsibility for what
• Select and collect meaningful work.
• The students periodically reflect in writing on
their portfolios
• Have other students, teachers, outsiders
periodically examined the portfolios.