H. DOUGLAS BROWN PRIYANVADA ABEYWICKRAMALanguage Assessment, Principles and Classroom Practices, Second Edition Copyright © 2010 by Pearson Education, lnc. All rights reserved. No part of this publication may be reproduced, stored ina retrieval system, or transmitted in any form (or by means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of e-publisher. ear BAG! til sures, White Pains, NY 10606 Staff credits: The people who made up the Language Assessment, Principles and Classroom Practices, Second Baition team, representing editorial, production, design, and manufacturing, are Danielle Belfiore, John Brezinsky, Tracey Munz Cataldo, Nancy Flaggman, Jaime Lieber, Lise Minoviz Kim Steiner, and jennifer Stem. Cover design: Tracey Munz Cataldo ‘Text design: Wendy Wolf, TS! Graphics ‘Text composition: TSI Graphics ‘Text font: Garamond Book ‘Text art: Don Martner ‘Tent credits: See page xiv. Library of Congress Cataloging in-Publication Data Brown, H. Douglas, 1941- ‘Language assessment: principles and classroom practices/H. Douglas Brown; Priyanvada ‘Abeywickrama. p.om. ISBN 0.13.8149313, I, Language and languagesStudy and teaching, 2. Language and languages— Examinations, 3. Language acquisition. 1. Abeywickrama, Priyanvada. I. Tite. P53.4.876 2010 418,0076—de22 2009039195 ISBN-13: 9780138149314 1SBN-10: 0138149313 PETE Pearsonlongman.com offers online resources for teachers and students. Access ‘our Companion Websites, our online catalog, ‘and our local offices around the world. Vist us at pearsonfongman.com. Printed in the United States of America 123456789 10—VoS6—15 14 13 12 11 10 CONTI Preface Text Credit Chapter 1 Assessment an Measu Assess Inform Forma Norm] Types and Pury ‘Achiev Diagno Placem Proficic Aptitue Issues in Langu Behave Integrat Commi Perfor Current “Hot Te Multiple ‘Traditio Comput Other ¢ Brercises 22 For Your Furthe Chapter 2 1 Practicality 26 Reliability 27 Student.CHAPTER 2 PRINCIPLES OF LANGUAGE ASSESSMENT OBJECTIVES: After reading this chapter, you will be able to * understand the five major * analyze the relative, variable Principles of language assessment importance of each principle, (practicality, reliability, validity, depending on the context and authenticity, and washback) and purpose of the assessment the essential subcategories within + apply each principle to classroom- reliability and validity based assessment instruments and * cite examples that support and/or ‘make an informed decision on the fail to support each principle ‘extent to which a principle is supported in a given instrument ‘This chapter explores how principles of language assessment can and should be applied to formal tests but with the ultimate recognition that these principles also apply to assessments of all kinds. In this chapter, these principles are defined and discussed with reference to classroom-based assessment in particular. They are then rephrased in the form of a set of “tips” for testing that can be applied to a number of different kinds of classroom assessments. Chapter 3 then focuses on using these principles, step-by-step, in the actual design of classroom-based tests. How do you know if a test is effective, appropriate, useful, or, in down-to- arth terms, a“good’ test? For the most part, that question can be answered by responding to such questions as: Can it be given within appropriate administra- tive constraints? Is it dependable? Does it accurately measure what you want it to measure? Is the language in the test representative of real-world language use? Does the test provide information that is useful for the learner? These questions help to identify five cardinal criteria for “testing a test”: practicality, reliability, validity, authenticity, and washback. We will look at each one here; however, because all five principles are context dependent, there is no priority order implied in the order of presentation26 crurret 2. Principles of Language Assessment PRACTICALITY Practicality refers to the logistical, down-to-earth, administrative issues involved in making, giving, and scoring an assessment instrument. These include “costs, the ‘amount of time it takes to construct and to administer, ease of scoring, and ease of interpreting/reporting the results" (Mousavi, 2009, p. 518). A test that fails to meet such criteria is impractical. Consider the following attributes of practicality A PRACTICAL TEST . . - + stays within budgetary limits + can be completed by the test-taker within appropriate time constraints + has clear directions for administration + appropriately utilizes available human resources + does not exceed available material resources + considers the time and effort involved for both design and scoring [A test of language proficiency that takes a student five hours to complete is impractical—it consumes more time than is available to accomplish its objective. A test that requires individual one-on-one proctoring is impractical for a group of sev- ral hundred test-takers and only a handful of examiners, A test that takes a few min- utes for a student to take and several hours for an examiner to evaluate is impractical for most classroom situations. A test that can be scored only by computer is imprac- tical if the test takes place a thousand miles away from the nearest computer. A test that relies too heavily on the subjective “hunches” of the scorer might be impractical (as well as unreliable) because it takes too long to score. The value and quality of a test sometimes hinge on such nitty-gritty, practical considerations. Here's a little horror story about practicality gone awry. An administrator of a six-week summertime short course needed to place the 50 or so students who had enrolled in the program. A quick search yielded a copy of an old English Placement ‘Test from the University of Michigan, It had 20 listening items based on an audio- tape and 80 items on grammar, vocabulary, and reading comprehension, all multiple- choice format. A scoring grid accompanied the test. On the day of the test, the required number of test booklets had been secured, a proctor had been assigned to monitor the process, and the administrator and proctor had planned to have the scoring completed by later that afternoon so students could begin classes the next day. Sounds simple, right? Wrong. ‘The students arrived, test booklets were distributed, and dircctions were given, The proctor started the audio recording. Soon students began to look puz led, By the time the tenth item played, everyone looked bewildered. Finally, the proctor checked a test booklet and was shocked to discover that the wrong audio ‘program was playing; it contained items for another form of the same test! Now what? She decided to randomly select a short passage from a textbook that was in the room and give the students a dictation. The students responded reasonably well, The the studen When the tests, t process th exchange, been score ack to the ‘The tv items whe: Students w evening, ha the two fru It's eas tion of the 1 scored), th as you will proctor anc time constr cality factor RELIABILITY A reliable | student or t ilar results." A RELL + is cons * gives c + has uni * Jends it * contain The issu number of fu sible factors 1 administratio: Fulcher & Da some of whicCoupree 2 Principles of Language Assesment 27 well. The subsequent 80 nontape-based items proceeded without incident, and the students handed in their score sheets and dictation papers. ‘When the red-faced administrator and the proctor got together later t0 score the tests, they faced the problem of how to score the dictation—a more subjective process than some other forms of assessment (sce Chapter 7). After a lengthy ‘exchange, the two established a point system, but after the first few papers had been scored, it was clear that the point system needed revision. That meant going back to the first papers to make suré the new system was followed. ‘The two faculty members had barely begun to score the 80 multiple-choice items when students began returning to the office to receive their placements. Students were told to come back the next morning for their results. Later that evening, having combined dictation scores and the 80item multiple-choice scores, the two frustrated examiners finally arrived at placements for all students, It's easy to sce what went wrong here. While the listening comprehension sec- tion of the test was apparently highly practical (easily administered and very quickly scored), the administrator had failed to check the materials ahead of time (which, as you will sec later, is a factor that touches on unreliability as well). Then the proctor and administrator established a scoring procedure that did not fit into the time constraints. In classroom-based testing, time is almost always a crucial practi cality factor for busy teachers with too few hours in the day. RELIABILITY. A reliable test is consistent and dependable. If you give the same test to the same student or matched students on two different occasions, the test should yield sim- ilar results. We might capsulate the principle of reliability in the following: A RELIABLE TEST... + is consistent in its conditions across two or more administrations + gives clear directions for scoring/evaluation + has uniform rubrics for scoring/evaluation + lends itself to consistent application of those rubrics by the scorer + contains items/tasks that are unambiguous to the testtaker The issue of the reliability of tests can be better understood by considering a ‘number of factors that can contribute to their unreliability. We look here at four pos- sible factors regarding fluctuations in (a) the student, (b) the scoring, (c) the test administration, and (d) the test itself. (See Bachman, 1990, J. D. Brown, 2005, and Fulcher & Davidson, 2007, for further and more elaborated discussions of reliability, some of which extend well beyond teacher-made classroom assessments.)28 cinereR 2 Principles of Language Assessment Student Related Reliability “The most common learner-related issue in reliability is caused by temporary illness, fatigue, a*bad day? anxiety, and other physical or psychological factors, which may ‘make an observed score deviate from one's “true” score. Also included in this category are such factors as a testtaker's test-wiseness, or strategies for efficient test-taking (Mousavi, 2009, p. 804), For the classroom teacher, student-related unreliability may at first blush seem to be a factor beyond control, We're accustomed to simply expecting some students to be anxious or overly nervous to the point that they “choke” in a test administration context. But the experience of many teachers suggests otherwise. In the second half of this chapter, some tips will be offered that may help mini- mize studentrelated unreliability. Rater Reliability ‘Human error, subjectivity, and bias may enter into the scoring process. Inter-rater reliability occurs when two or more scorers yield consistent scores of the same test, Failure to achieve intra-rater reliability could stem from lack of adherence to scoring criteria, inexperience, inattention, or even preconceived biases. Lumley (2002) provided some helpful hints on how to ensure inter-rater reliability. Raterteliability issues are not limited to contexts in which two or more scorers are involved. Intra-rater reliability is an internal factor, a common occur- rence for classroom teachers. Violation of such reliability can occur in cases of unclear scoring criteria, fatigue, bias toward particular “good” and “bad” students, or simple carelessness, When I am faced with up to 40 essay tests (for which there is no absolute right or wrong set of answers) to grade in only a week, I know that the standards | apply —however subliminally—to the first few tests will be different from those I apply to the last few. I may be “easier” or “harder” on those first few papers or I may get tired, and the result may be an inconsistent evaluation across all tests, One solution to such intraater unreliability is to read through about half of the tests before rendering any final scores or grades, then to recycle back through the whole set of tests to ensure an evenchanded judgment. In tests of writing skills, rater reliability is particularly hard to achieve because writing profi ciency involves numerous traits that are difficult to define. The careful specifica- tion of an analytical scoring instrument, however, can increase both inter- and intra-rater reliability J. D. Brown, 1991). ‘Test Administration Reliability Unreliability may also result from the conditions in which the test is administered, Tonce witnessed the administration of a test of aural comprehension in which an audio player was used to deliver items for comprehension, but because of strect noise outsi: the stimuli of the test : variations, 1 ture, and ev Test Reliabilit Sometimes ‘multiple-ch istics that « difficult, dis to make the because the teacher-mac tical perspe and Fulchet In clas: tors, includ ended resp the teacher trast, have their test re Furthe: that are amt tains too m: may ultimat items and hi ‘who do not may be incl ‘who are ady it is obvious muddying t1 VALIDITY By far the m tant principl results are a ment” (Gron] (1989), who grated evaluaoumreR 2. Principles of Language Assessment 29 noise outside the building, students sitting next to open windows could not hear the stimuli accurately. This was a clear case of unreliability caused by the conditions of the test administration, Other sources of unreliability are found in photocopying variations, the amount of light in different parts of the room, variations in tempera ture, and even the condition of desks and chairs, ‘Test Reliability Sometimes the nature of the test itself can cause measurement errors. Tests with multiple-choice items must be carefully designed to include a number of character- istics that will guard against unreliability. For example, the items need to be evenly difficult, distractors need to be well designed, and items need to be well distributed to make the test reliable. In this book, these forms of reliability are not discussed because they rarely are appropriately applied to classroom-based assessment and teacher-made tests. (For a full discussion of reliability from a psychometric, statis- tical perspective, consult the aforementioned Bachman [1990], J. D. Brown (2005), and Fulcher & Davidson (2007].) In classroom-based assessment, test unreliability can be caused by many fac- tors, including rater bias. This typically occurs with subjective tests, with open- ended responses (e.g., essay responses) that require a judgment on the part of the teacher to determine correct and incorrect answers. Objective tests, in con- trast, have predetermined fixed responses, their test reliability Further unreliability may be caused by poorly written test items—that is, items that are ambiguous or that have more than one correct answer, Also, a test that con- tains too many items (beyond what is needed for discriminating among students) ‘may ultimately cause test-takers to become fatigued by the time they reach the later items and hastily respond incorrectly. Timed tests may discriminate against students ‘who do not perform well on a test with a time limit. We all know people (and you may be included in this category) who “know” the course material perfectly but who are adversely affected by the presence of a clock ticking away. In such cases, it is obvious that test characteristics can interact with studentrelated unreliability, muddying the lines of distinction between the two types. a format which of course increases VALIDITY By far the most complex criterion of an effective test—and arguably the most impor- tant principle—is validity, “the extent to which inferences made from assessment results are appropriate, meaningful, and useful in terms of the purpose of the assess- ment” (Gronlund, 1998, p. 226). In somewhat more technical terms, Samuel Messick. (1989), who is widely recognized as an expert on validity, defined validity as“an inte- grated evaluative judgment of the degree to which empirical evidence and theoretical30 2 Principles of Language Assessment rationales support the adequacy and appropriateness of inferences and actions based ‘on test scores or other modes of assessment” (p. 11), ‘We might infer from these definitions the following attributes of validity: A VALID TEST... + measures exactly what it proposes to measure + does not measure irrelevant of “contaminating” variables + relies as much as possible on empirical evidence (performance) + involves performance that samples the test’s criterion (objective) + offers useful, meaningful information about a testtaker’s ability + is supported by a theoretical rationale or argument [A valid test of reading ability actually measures reading ability—not 20/20 vision, or previous knowiedge of a subject, or some other variable of questionable relevance. To measure writing ability, one might ask students to write as many ‘words as they can in 15 minutes, then simply count the words for the final score. Such a test would be easy to administer (practical), and the scoring quite depend- able (reliable), but it would not constitute a valid test of writing ability without some consideration of comprehensibility, rhetorical discourse elements, and the organization of ideas, among other factors. How is the validity of a test established? According to Weir (2005), Broadfoot (2005), and‘. McNamara (2006), there is no final, absolute measure of valiity, but several different kinds of evidence may be invoked in support. Moreover, as Messick (1.989) emphasized, it is important to note that validity isa matter of degree, not all or none” (p. 33). Tn some cases, it may be appropriate to examine the extent to which atest calls for performance that matches that of the course or unit being tested. In other cases, ‘we may be concerned with how well a test determines Whether students have reached an established set of goals or level of competence. Statistical correlation with other related but independent measures is another widely accepted form of evidence. Other concems about a tests validity may focus on the consequences of 2 test, beyond measuring the criteria themselves, or even on the testtaker's percep- tion of validity. We look at four types of evidence below. Content Related Evidence If test actually samples the subject matter about which conclusions are to be drawn, and if it requires the testtaker to perform the behavior that is being mez- sured, it can claim content-related evidence of validity, often popularly referred to as content-related validity (c.g, Hughes, 2003; Mousavi, 2009). You can usually identify contentelated evidence observationally if you can clearly define the achievement that you are measuring. A test of tennis competency that asks someone to run a 10 person's abi earner to at judgments ally speak w 10 objective Consid: conversatior Directions apply the article) in Last night You know, Francisco as so wo wal monkey st ‘The stu discussions and speakin it uses a fan fact that itv read the pas fora listenir There a ments that 9 to contend, textreduced lack content tive perform sion). There proficiency evidence, ne Anothe: between dir ally perform task itself buchaprER 2 Principles of Language Assessment 31 to run a 100-yard dash obviously lacks content validity. If you are trying to assess a person's ability to speak a second language in a conversational setting, asking the learner to answer paper-and-pencil multiple-choice questions requiring grammatical judgments does not achieve content validity. A test that requires the learner to actu ally speak within some sort of authentic context does. And if a course has perhaps 10 objectives but only two are covered in a test, then content validity suffers. Consider the following quiz on English articles for a high-beginner level of a conversation class (listening and speakinj) for English-learners, Directions: The purpose ofthis quiz is for you and me to find out how well you know and can. apply the rules of article usage. Read the following passage and write a/an, the, or 0 (no article) in each blank. Last night, | had (1) very strange dream. Actually, it was (2) nightmare! ‘You know how much t ove (3) ‘7008. Wel, | dreamt that | went to (4) ‘San, Francisco 200 with (5) ‘ew friends. When we got there, It was very dark, but © ‘moon was out, 80 we weren't afaid | wanted to see (7) monkeys frst, ‘80 we walked past (8) merry-go-round and (9) lions? cages to (10) monkey section ‘The students had had a unit on 200 animals and had engaged in some open discussions and group work in which they had practiced articles, all in listening and speaking modes of performance. This quiz is somewhat content valid because it uses a familiar setting and focuses on previously practiced language forms. The fact that it was administered in written form, however, and required students to read the passage and write their responses makes it quite low in content validity fora listening/speaking class There are a few cases of highly specialized and sophisticated testing instru- ‘ments that may have questionable content-elated evidence of validity. It s possible to contend, for example, that standard language proficiency tests, with their con- textreduced, academically oriented language and limited stretches of discourse, lack content validity because they do not require the fall spectrum of communica. tive performance on the part of the learner (See Bachman, 1990, for a full discus- sion). There is good reasoning behind such criticism; nevertheless, what such proficiency tests lack in content-elated evidence they may gain in other forms of evidence, not to mention practicality and reliability. ‘Another way of understanding content validity is to consider the difference between direct and indirect testing, Direct testing involves the testtaker in actu. ally performing the target task. In an indirect test, learners are not performing the task itself bt rather a task that is related in some way. For example, if you intend to32 ciupren 2 Principles of Language Assessment test learners’ oral production of syllable stress and your test task is to have learners mark (with written accent marks) stressed syllables in a list of written words, you could, with a stretch of logic, argue that you are indirectly testing their oral produc- tion. A direct test of syllable production would require that students actually pro- duce target words orally ‘The most feasible rule of thumb for achieving content validity in classroom assessment is to test performance directly. Consider, for example, a listening/ speaking class that is doing a unit cin greetings and exchanges that includes discourse for asking for personal information (name, address, hobbies, etc.) with some form- focus on the verb be, personal pronouns, and question formation, The test on that unit should include all of the above discourse and grammatical elements and involve students in the actual performance of listening and speaking, ‘What all the above examples suggest is that content is not the only type of evidence to support the validity of a test; in addition, classroom teachers have nei- ther the time nor the budget to subject quizzes, midterms, and final exams to the extensive scrutiny of a full construct validation (see the next page). Therefore, teachers should make every effort to ensure that their tests have content validity by examining the extent to which the tests assess real course objectives and by ‘ensuring that they test directly. Criterion-Related Evidence ‘A second form of evidence of the validity of a test may be found in what is called criterion-elated evidence, also referred to as criterion-related validity, or the extent to which the “criterion” of the test has actually been reached, You will recall from Chapter 1 that most classroomvbased assessment with teacherdesigned tests fits the concept of criterion-referenced assessment. In such tests, specified class- ‘room objectives are measured, and implied predetermined levels of performance are expected to be reached (80 percent is considered a minimal passing grade). In the case of teacher-made classroom assessments, criterion-telated evidence is best demonstrated through a comparison of results of an assessment with results of some other measure of the same criterion, For example, in a course unit whose objective is for students to be able to orally produce voiced and voiceless stops in all possible phonetic environments, the results of one teacher’s unit test might be ‘compared with an independent assessment—possibly a commercially produced test in a textbook—of the same phonemic proficiency. A classroom test designed to assess mastery of a point of grammar in communicative use will have criterion validity if test scores are corroborated either by observed subsequent behavior or by other communicative measures of the grammar point in question. Criterion-related evidence usually falls into one of two categories: concurrent and predictive validity. A test has concurrent validity if its results are supported by other concurrent performance beyond the assessment itself. For example, the validity of a high score on the final exam of a foreign language course will be sub- sstantiated by actual proficiency in the language. The predictive validity of an ment b to*moy sure cx future + Construct: Atthird arole fi constr to expk may no! ential d linguist Virwally structs. into the dent's 6 various other el operatic of the T.McNa For struct v perhaps don't le tion of \ Im oral inte final. scx sociolin oretical So if yor pronunc validity « ulary qu define a what we municat a constr Cor proficierueree 2 Principles of Language Assessment 33 assessment becomes important in the case of placement tests, admissions assess- ment batteries, and achievement tests designed to determine students’ readiness to"move on" to another unit. The assessment criterion in such cases is not to mea- sure concurrent ability but to assess (and predict) a testtaker's likelihood of future success, Construct -Related Evidence A third kind of evidence that can support validity, but one that does not play as large a role for classroom teachers, is construct-related validity, commonly referred to as construct validity. A construct is any theory, hypothesis, or model that attempts to explain observed phenomena in our universe of perceptions. Constructs may or may not be directly or empirically measured—their verification often requires infer- ential data. Proficiency, communtcative competence, and fluency are examples of linguistic constructs; seifesteem and motivation are psychological constructs Virtually every issue in language learning and teaching involves theoretical con- structs. In the field of assessment, construct validity asks, "Does this test actually tap into the theoretical construct as it has been defined?” Let's say you're assessing a stu: dent's oral fluency. To possess construct validity, your test should account for the various components of fluency: speed, rhythm, juncture, (lack of) hesitations, and other elements within the construct of fluency. Tests are, in a manner of speaking, operational definitions of constructs in that their test tasks are the building blocks of the entity that is being measured (see Davidson, Hudson, & Lynch, 1985; T. McNamara, 2000). For most of the tests that you administer as a classroom teacher, a formal con- struct validation procedure may seem a daunting prospect. You will be tempted, perhaps, to run a quick content check and be satisfied with the test's validity. But don't let the concept of construct validity scare you. An informal construct valida- tion of virtually every classroom test is both essential and feasible. Imagine, for example, that you have been given a procedure for conducting an oral interview. The scoring analysis for the interview includes several factors in the final score: pronunciation, fluency, grammatical accuracy, vocabulary use, and socio-linguistic appropriateness. The justification for these five factors lies in a the: retical construct that claims those factors as major components of oral proficiency. So if you were asked to conduct an oral proficiency interview that evaluated only Pronunciation and grammar, you could be justifiably suspicious about the construct validity of that test. Likewise, let's suppose you have created a simple written vocab- ulary quiz, covering the content of a recent unit, which asks students to correctly define a set of words. Your chosen items may be a perfectly adequate sample of what was covered in the unit, but if the lexical objective of the unit was the com- municative use of vocabulary, then the writing of definitions certainly fails to match a construct of communicative language use. Construct validity is a major issue in validating large-scale standardized tests of proficiency. Because such tests must, for economic reasons, adhere to the principle34 ciupren 2 Principles of Language Assessment of practicality, and because they must sample a limited number of domains of lan- guage, they may not be able to contain all the content of a particular field or skill. For example, many large-scale standardized tests worldwide have until recently not attempted to sample oral production, yet oral production is obviously an important aspect of language ability. The omission of oral production content, however, was ostensibly justified by research that showed positive correlations between oral pro- duction and the behaviors (listening, reading, grammaticality detection, and writing) actually sampled on stich tests (Duran, Canale, Penfield, Stansfield, & Liskin-Gasparo, 1985). Because of the crucial need to offer financially affordable proficiency tests and the high cost of administering and scoring oral production tests, the omission of oral content was justified as an economic necessity. However, in the last decade, with advances in developing rubrics for scoring oral production tasks and in automated speech recognition software, more general language profi ciency tests include oral production tasks, largely stemming from the demands of the professional community for authenticity and content validity. Consequential Validity (impact) ‘As well as these three widely accepted forms of evidence that may be introduced to support the validity of an assessment, two other categories may be of some interest and utility in your own quest for validating classroom tests. Messick (1989), ‘T. McNamara (2000), Brindley (2001), Fulcher and Davidson (2007), and Gronlund and Waugh (2008), among others, underscore the potential importance of the con- sequences of using an assessment. Consequential validity encompasses all the ‘consequences of a test, including such considerations as its accuracy in measuring intended criteria, its effect on the preparation of testtakers, and the (intended and unintended) social consequences of a test's interpretation and use. Bachman and Palmer (1996), McKay (2000), Davies (2003), and Choi (2008) use the term impact to refer to consequential validity, perhaps more broadly encom- passing the many consequences of assessment, before and after a test administration. ‘The impact of test-taking and the use of test scores can, according to Bachman and Palmer (p. 30), be seen at both a macro level (the effect on society and educational systems) and a micro level (the effect on individual test-takers). At the macro level, Choi argued that the wholesale employment of standardized tests for such gate: keeping purposes as college admission “deprive students of crucial opportunities to earn and acquire productive language skills” causing test consumers to be “increas- ingly disillusioned with EFL testing” (p. 58). More will be said about impact and related issues of values, social consequences, ethics, and faimess in Chapter 4 [As high-stakes assessment has gained ground in the last two decades, one aspect of consequential validity has drawn special attention: the effect of test prepa- ration courses and manuals on performance. T. McNamara (2000) cautioned against test results that may reflect socioeconomic conditions such as opportunities for coaching that are “differentially available to the students being assessed (for example more hi Att tant con more fu teachers perform school ¥ Face Validi A furthe P. 210), validity 1 knowlec examine other ps Des that can, validity. sibly the ment ex factor th mortem that face cian's ve Att agree th test desi testing ¥ sequent ception ‘can affes tion of £ sgeeneCompree 2. Principles of Language Assessment 35 ‘example, because only some families can afford coaching, or because children with more highly educated parents get help from their parents)” (p. 54) ‘Atthe micro level, specifically the classroom instructional level, another impor- tant consequence of a test falls into the category of washback, to be defined and more fully discussed later in this chapter. Gronlund and Waugh (2008) encouraged teachers to consider the effect of assessments on students’ motivation, subsequent performance in a course, independent learning, study habits, and attitude toward school work. Face Validity AA further facet of consequential validity is the extent to which ‘students view the assessment 5 fair, relevant, and useful for improving learning” (Gronlund, 1998, p. 210), or what has popularly been called—or misnamed—face validity. “Face validity refers to the degree to which a test ooks right, and appears to measure the knowledge or abilities it claims to measure, based on the subjective judgment of the examinees who take it, the administrative personnel who decide on its use, and other psychometrically unsophisticated observers" (Mousavi, 2009, p. 247). Despite the intuitive appeal of the concept of face validity, it remains a notion that cannot be empirically measured or theoretically justified under the category of validity. It is purely a factor of the “eye of the beholder” —how the testtaker, or pos- sibly the test giver, intuitively perceives an instrument. For this reason, many assess- ‘ment experts (See Bachman, 1990, pp. 285-289) view face validity as a superficial factor that is too dependent on the whim of the perceiver. In Bachman's “post- mortem” on face validity, he echoes Mosier’s (1947, p. 194) decades-old contention that face validity is a“pernicious fallacy ... [that should be] purged from the techni- ian’s vocabulary” At the same time, Bachman (1990) and other assessment experts “grudgingly agree that test appearance does indeed have an effect that neither testtakers nor test designers can ignore. Students may for a variety of reasons feel that a test isn't testing what it's supposed to test, and this might affect their performance and, con- sequently, create student-related unreliability referred to previously. So student per- ception of a test's fairness is significant to classroom-based assessment because it ‘can affect student performance/reliability. Teachers can increase student's percep: tion of fair tests by using + a well-constructed, expected format with familiar tasks «+ tasks that can be accomplished within an allotted time limit + items that are clear and uncomplicated * directions that are crystal clear + tasks that have been rehearsed in their previous course work + tasks that relate to their course work (content validity) a difficulty level that presents a reasonable challenge36 Curren 2 Principles of Language Assessment Finally, the issue of face validity reminds us that the psychological state of the learner (confidence, anxiety, etc.) is an important ingredient in peak performance. Students can be distracted and their anxiety increased if you “throw a curve” at them on a test. They need to have rehearsed test tasks before the fact and feel comfort- able with them. A classroom test is not the time to introduce new tasks, because you won't know if student difficulty is a factor of the task itself or of the objectives you are testing T once administered a dictation test and a cloze test (see Chapter 9 for a dis- cussion of cloze tests) as a placement test for a group of learners of English as a second language. Some learners were upset because such tests, on the face of it, did not appear to them to test their true abilities in English. They felt that a multiple- choice grammar test would have been the appropriate format to use. A few claimed they didn’t perform well on the cloze and dictation because they were not accus- tomed to these formats. As it turned out, the tests served as superior instruments for placement, but the students did not think so. As already noted above, validity is a complex concept, yet it is indispensable to the teacher's understanding of what makes a good test. We do well to heed Messick’s 1989, p. 33) caution that validity is not an allornone proposition and that various forms of validity may need to be applied to a test in order to be satisfied with its overall effectiveness. If in your language assessment procedures you can make a point of primarity focusing on content and criterion validity, then you are well on your way to making accurate judgments about the competence of the learners with ‘whom you are working, AUTHENTICITY ‘A fourth major principle of language testing is authenticity, a concept that is diffi cult to define, especially within the art and science of evaluating and designing tests. Bachman and Palmer (1996) defined authenticity as “the degree of correspon- dence of the characteristics of a given language test task to the features of a target language task” (p. 23) and then suggested an agenda for identifying those target lan- ‘guage tasks and for transforming them into valid test items. ‘As mentioned, authenticity is not a concept that easily lends itself to empir- ical definition or measurement. (Lewkowicz [2000] discussed the difficulties of operationalizing authenticity in language assessment.) After all, who can certify whether a task or language sample is “real-world” or not? Often such judgments are subjective, and yet authenticity is a concept that language-testing experts have paid a great deal of attention to (Bachman & Palmer, 1996; Fulcher & Davidson, 2007). Further, according to Chun (2006), many test types fail to simulate real- ‘world tasks. Essentially, when you make a claim for authenticity in a test task, you are saying that this task is likely to be enacted in the real world. Many test item types fail to simulate + target ag relationsh to find re reakworld Inat AN AL + cont + has inch * prov stor + offe: The a three dec: sary comy scale testi budgetary nents. Re: likely to h: ture natur offer item: paragraph We in As we ext through 9 WASHBACK A facet of and learni field as w: from wast issues sucl developm« refer to be may be ret Wall (199 Washbackoupree 2 Principles of Language Assessment 37 simulate real-world tasks. They may be contrived or artificial in their attempt to target a grammatical form or a lexical item. The sequencing of items that bear no relationship to one another lacks authenticity. One does not have to look very long, to find reading comprehension passages in proficiency tests that do not reflect a real-world passage, In a test, authenticity may be present in the following ways: AN AUTHENTIC TEST... * contains language that is as natural as possible + has items that are contextualized rather than isolated + includes meaningful, relevant, interesting topics + provides some thematic organization to items, such as through a story line or episode + offers tasks that replicate real-world tasks ‘The authenticity of test tasks in recent years has increased noticeably. Two ot three decades ago, unconnected, boring, contrived items were accepted as a neces sary component of testing. Things have changed. It was once assumed that large scale testing could not include performance of the productive skills and stay within budgetary constraints, but now many such tests offer speaking and writing compo- nents, Reading passages are selected from real-world sources that testtakers are likely to have encountered or will encounter. Listening comprehension sections fea- ‘ure natural language with hesitations, white noise, and interruptions, More tests offer items that are “episodic” in that they are sequenced to form meaningful units, paragraphs, or stories. We invite you to take up the challenge of authenticity in your classroom tests. 4s we explore many different types of tasks in this book, especially in Chapters 6 through 9, the principle of authenticity will be very much in the forefront. WASHBACK A facet of consequential validity discussed above is “the effect of testing on teaching and learning” (Hughes, 2003, p. 1), otherwise known in the language assessment field as washback. To distinguish the impact of an assessment, discussed above, from washback, think of the latter as referring almost always to classroom-based issues such as the extent to which assessment affects a student's future language development. Messick (1996, p. 241) reminded us that the washback effect may refer to both the promotion and the inhibition of learning, thus emphasizing what may be referred to as beneficial versus harmful (or negative) washback. Alderson & Wall (1993) considered washback an important enough concept to define a Washback Hypothesis that essentially elaborated on how tests influence both38 ciueren 2 Principles of Language Assessment teaching and learning. Cheng, Watanabe, and Curtis (2004) devoted an entire anthology to the issue of washback, and Spratt (2005) challenged teachers to become agents of beneficial washback in their language classrooms. ‘The following factors comprise the concept of washback A TEST THAT PROVIDES BENEFICIAL WASHBACK . . + positively influences what aid how teachers teach + positively influences what and how learners learn * offers learners a chance to adequately prepare + gives learners feedback that enhances their language development + is more formative in nature than summative * provides conditions for peak performance by the learner In lange-scale assessment, washback often refers to the effects that tests have on instruction in terms of how students prepare for the test. “Cram” courses and “teaching to the test” are examples of washback that may have both negative and pos- itive effects. The current worldwide use of standardized tests for gate-keeping pur- poses can Jead students to focus on simply gaining an acceptable score rather than. on language development. On the positive side, many enrollees in testpreparation courses report increased competence in certain languagerelated tasks (Chapelle, Enright, & Jamieson, 2008). Tn classroom-based assessment, washback can have a number of positive man- ifestations, ranging from the benefit of preparing and reviewing for a test to the learning that accrues from feedback on one’s performance. Teachers can provide information that “washes back" to students in the form of useful diagnoses of strengths and weaknesses. Washback also includes the effects of an assessment on teaching and learning prior to the assessment itself, that is, on preparation for the assessment. Informal performance assessment is by nature more likely to have built- in washback effects because the teacher is usually providing interactive feedback. Formal tests can also have positive washback, but they provide no beneficial wash- back if the students receive a simple letter grade or a single overall numerical score. ‘The challenge to teachers is to create classroom tests that serve as learning devices through which washback is achieved. Students’ incorrect responses can become windows of insight into further work. Their correct responses need to be praised, especially when they represent accomplishments in a student's developing language competence. Teachers can suggest strategies for success as part of their “coaching” role. Washback enhances a number of basic principles of language acqui- sition: intrinsic motivation, autonomy, self-confidence, language ego, interlanguage, and strategic investment, among others. (See PLT and TBP for an explanation of these principles.) ‘One way to enhance washback is to comment generously and specifically on test. performance. Many overworked (and underpaid) teachers return tests to students with a sir letter gra to the st. reduce th nothing. ‘mance as tive, learn with productic back. Eve can respe for streng Give strat mance. Ir cally mo accompli: A litt numerica tenses, fo pose of s1 Anot ferencé Formative learner 01 mative te: need to 0! the end o, suits, mor tion inac Inm session, 1 plete the: class. Att] and I spe were not beneficial Finall feedback teachers ¥ ative, coll: dents anc students r of any isst the days aComores 2. Principles of Language Assesment 39 with a single letter grade or numerical score and consider their job done. In reality, letter grades and numerical scores give absolutely no information of intrinsic interest to the student. Grades and scores alone, without comments and other feedback, reduce the linguistic and cognitive performance data available to student to almost nothing. At best, they give a relative indication of a formulaic judgment of perfor. ‘mance as compared to others in the class—which fosters competitive, not coopera- tive, learning, With this in mind, when you return a written test or a data sheet from an oral production test, consider giving more than a number, grade, or phrase as your feed- back. Even if your evaluation is not a neat little paragraph appended to the test, you can respond to as many details throughout the test as time will permit. Give praise for strengths—the “good stuff"—as well as constructive criticism of weaknesses, Give strategic hints on how a student might improve certain elements of perfor- mance. In other words, take some time to make the test performance an intrinsi- cally motivating experience from which a student will gain a sense of accomplishment and challenge. Avlittle bit of washback may also help students through a specification of the numerical scores on the various subsections of the test. A subsection on verb tenses, for example, that yields a relatively low score may serve the diagnostic pur pose of showing the student an area of challenge. Another viewpoint on washback is achieved by a quick consideration of dif: ferences between formative and summative tests, mentioned in Chapter 1 Formative tests, by definition, provide washback in the form of information to the earner on progress toward goals. But teachers might be tempted to feel that sum- mative tests, which provide assessment at the end of a course or program, do not need to offer much in the way of washback. Such an attitude is unfortunate because the end of every language course or program is always the beginning of further pur- Suits, more learning, more goals, and more challenges to face. Even a final examina tion in a course should carry with it some means for giving washback to students. In my courses I never give a final examination as the last scheduled classroom session. | always administer a final exam during the penultimate session then com- plete the evaluation of the exams in order to return them to students during the last lass. At this time, the students receive scores, grades, and comments on their work, and I spend some of the class session addressing material on which the students were not completely clear. My summative assessment is thereby enhanced by some beneficial washback that is usually not expected of final examinations, Finally, washback implies that students have ready access to you to discuss the feedback and evaluation you have given, Whereas you almost certainly have known teachers with whom you wouldn't dare argue about a grade, an interactive, cooper: ative, collaborative classroom can promote an atmosphere of dialogue between stu dents and teachers regarding evaluative judgments. For learning to continue, students need to have a chance to feed back on your feedback, to seek clarification of any issues that are fuzzy, and to set new and appropriate goals for themselves for the days and weeks ahead40 coupren2 Principles of Language Assessment APPLYING PRINCIPLES TO THE EVALUATION OF CLASSROOM TESTS ‘The ive principles of practicality, reliability, validity, authenticity, and washback go a long way toward providing useful guidelines for both evaluating an existing assess- ‘ment procedure and designing one on your own. Quizzes, tests, final exams, and standardized proficiency tests can all be scrutinized through these five lenses. ‘Are there other principles that should be invoked in evaluating and designing, assessments? The answer, of course, is yes. Language assessment is an extraordinarily broad discipline with many branches, interest areas, and issues. The process of designing effective assessment instruments is far too complex to be reduced to five principles. Good test construction, for example, is governed by research-based rules of test preparation, sampling of tasks, item design and construction, scoring responses, ‘ethical standards, and so on. But the five principles cited here serve as an excellent foundation on which to evaluate existing instruments and to build your own. ‘We will look at how to design tests in Chapter 3. The tips and checklists that follow in this chapter, indexed by the five principles, will help you evaluate existing tests for your own classroom. It is important for you to remember, however, that the sequence of these questions does not imply a priority order. Validity, for example, is certainly the most significant cardinal principle of assessment evaluation. Practicality could be a secondary issue in classroom testing. Or, for a particular test, you may need to place authenticity as your primary consideration. When all is said and done, however, if validity is not substantiated, all other considerations may be rendered useless. 1. Are the test procedures practical? Practicality is determined by the teacher's (and the students’) time constraints, costs, and administrative details and to some extent by what occurs before and after the test, To determine whether a test is practical for your needs, you may want to use the checklist below. ov PRACTICALITY CHECKLIST |. Are administrative details all carefully attended to before the test? . Can students complete the test reasonably within the set time frame? . Can the test be administered smoothly, without procedural “gltches"? . Are all printed materials accounted for? 5. Has equipment been pre-tested? 5. Is the Cost of the test within budgeted limits? Is the scoring/evaluation system feasible in the teacher's time frame? ‘Are methods for reporting results determined in advance? | a ooooocoo rrr As this, giving a test test. In teac that overrid atest to fity this withou example, a¥ tion items 1 teachers sec and will do possible. Ye time in givit 2. Is the test i Reliability ap ‘At least four on pages 27. that all stude owing chee 3. Can you en Rater reliabi hhaps becaus involve two ability is of tion and sta ‘Teachers nev to score ass importance. required to «crurren 2. Principles of Language Assessment 41 As this checklist suggests, after you account for the administrative details of giving a test, you need to think about the practicality of your plans for scoring the test. In teachers’ busy lives, time often emerges as the most important factor, one | that overrides other considerations in evaluating an assessment. If you need to tailor A test to fit your own time frame, as teachers frequently do, you need to accomplish this without damaging the test's validity and washback. Teachers should, for example, avoid the temptation to offer only quickly scored multiple-choice sele: tion items that may be neither appropriate nor welldesigned. Everyone knows | teachers secretly hate to grade tests (almost as much as students hate to take them) and will do almost anything to get through that task as quickly and effortlessly as possible, Yet good teaching almost always implies an investment of the teacher's time in giving feedback—comments and suggestions—to students on their tests. 2. Is the test itself reliable? } Reliability applies to the student, the test administration, the test itself, and the teacher. Atleast four sources of unreliability must be guarded against, as noted in this chapter on pages 27-29. Test and test administration reliability can be achieved by making sure that all students receive the same quality of input, whether written or auditory. The fol lowing checklist should help you to determine if atest is itself reliable: VU TEST RELIABILITY CHECKLIST 1. Does every student have a cleanly photocopied test sheet? 2. 1s sound amplification clearly audible to everyone in the room? 3. Is video input clearly and uniformly visible to all? 4. Are lighting, cemperature, extraneous noise, and other classroom Conditions equal (and optimal) for all students? 5. For closed-ended responses, do scoring procedures leave little debate about correctness of an answer? o oooo cc equ 3. Can you ensure rater reliability? Rater reliability, another common issue in assessments, may be more difficult, per haps because we too often overlook this as an issue. Because classroom tests rarely involve two scorers, inter-rater reliability is seldom an issue. Instead, intra-rater reli. | ability is of constant concern to teachers: What happens to our fallible concentra: tion and stamina over the period of time during which we are evaluating a test? ‘Teachers need to find ways to maintain their focus and energy over the time it takes to score assessments. In open-ended response tests, this issue is of paramount importance. It is easy to let mentally established standards erode over the hours required to evaluate the test.42 curren 2 Principles of Language Assessrent Intraater reliability for open-ended responses may be enhanced by answering these questions: U/ PRACTICALITY CHECKLIST Q a a Q a a TT Ot ITT 1. Have you established consistent criteria for correct responses? 2, Can you give uniform attention to those criteria throughout the ‘evaluation time? 3, Can you guarantee that scoring is based only on the established criteria, and not on extraneous of subjective variables? 4, Haye you read through tests at Ieast twice to check for consistency? 5. Ifyou have made “midstream” modifications of what you consider a correct response, did you go back and apply the same standards to all? 6. Can you avoid fatigue by reading the tests in several sittings, especially if the time requirement is a matter of several hours? 4. Does the procedure demonstrate content validity? ‘The major source for establishing validity in a classroom test is content validity: the extent to which the assessment requires students to perform tasks that were included in the previous classroom lessons and that directly represent the objectives of the unit on which the assessment is based. If you have been teaching an English language class to students who have been reading, summarizing, and responding to short passages, and if your assessment is based on this work, then to be content valid, the test needs to include performance in those skills. For classroom assessments, content and criterion validity are closely linked, because lesson or unit objectives are essentially the criterion of an assessment covering that lesson or unit. Several steps might be taken to evaluate the content validity of a classroom test: o CONTENT VALIDITY CHECKLIST (FOR A TEST oo oo o ON A UNIT) 1. Are unit objectives clearly identified? 2. Are unit objectives represented in the form of test specifications? (See the next page for details on test specifications.) 3. Do the test specifications include tasks that have already been ‘performed as part of the course procedures? 4, Do the test specifications include tasks that represent all (or most) of the objectives for the unit? 5. Do those tasks involve actual performance of the target task(6)? Aprim every good course in qu fication of 0 work throug they seek to mining whet A secon ‘word scare y ically from th + divide object * offers * gives: Some te ina course ir in-class writt manner of sf itself and in t return to the ‘The con the objective items, clustes test-takers as content valid 5. Has the imp This questior importance « mance of the appearanceCuupreR 2. Principles of Language Assessment 43 A primary issue in establishing content validity is recognizing that underlying every good classroom test are the objectives of the lesson, module, or unit of the course in question. So the first measure of an effective classroom test is the identi- fication of objectives. Sometimes this is easier said than done. Too often teachers work through lessons day after day with little or no cognizance of the objectives they seck to fulfill. Or perhaps those objectives are so poorly defined that deter mining whether they were accomplished is impossible. A second issue in content validity is test specifications (specs). Don't let this ‘word scare you. It simply means that a test should have a structure that follows log- ically from the lesson or unit you are testing. Many tests have a design that + divides them into a number of sections (corresponding, perhaps, to the objectives that are being assessed) * offers students a variety of item types * gives an appropriate relative weight to each section Some tests, of course, do not lend themselves to this kind of structure. A test in a course in academic writing at the university level might justifiably consist of an inclass written essay on a given topic—only one “item” and one response, in a ‘manner of speaking. But in this case the specs would be embedded in the prompt itself and in the scoring or evaluation rubric used to grade it and give feedback. We return to the concept of test specs in the next chapter. The content validity of an existing classroom test should be apparent in how the objectives of the unit being tested are represented in the form of the content of items, clusters of items, and item types. Do you clearly perceive the performance of testtakers as reflective of the classroom objectives? If so (and you can argue this), content validity has most likely been achieved. 5. Has the impact of the test been carefully accounted for? This question integrates the concept of consequential validity (impact) and the importance of structuring an assessment procedure to elicit the optimal perfor- ‘mance of the student. Remember that even though it is an elusive concept, the appearance of a test from a student’s point of view is important to consider.44° cimeren 2 Principles of Language Assessment The following factors might help you to pinpoint some of the issues sur- rounding the impact of a test: - CONSEQUENTIAL VALIDITY CHECKLIST 11. Have you offered students appropriate review and preparation for the test? 2, Have you suggested test-taking strategies that will be benefici / 3. Is the test structured so that, if possible, the best students will be I modestly challenged and the weaker students will not be overwhelmed? 4. Does the test lend itself to your giving beneficial washback? 4. Are the students encouraged to see the test as a learning experience? SS i a eal 6. 1s the procedure “biased for best”? ‘A phrase that has come to be associated with consequential validity is “biased for best,’ a term that goes a little beyond how the student views the test to a degree of strategic involvement on the part of student and teacher in preparing for, setting up, and following up on the test itself. According to Swain (1984), to give an assessment procedure that is biased for best, a teacher provides conditions for a student's optimal performance. In such a case, your role is not to be “tricky” or to scare your students but to encourage them and bring out the best in their performance. Cohen (2006) supported Swain's concept in a comprehensive discussion of research that showed the positive effects of students’ awareness and utilization of test-taking strategies. It’s easy for teachers to forget how challenging some tests can be, and soa well planned testing experience will include some strategic suggestions on how students might optimize their performance. In evaluating a classroom test, consider the extent to which before, during., and aftertest options are fulfilled. 7. Are the test tasks as authentic as possible? Evaluate the extent to which a testis authentic by asking the following questions: of AUTHENTICITY CHECKLIST }a Is the language in the test as natural as possible? Are items as contextualized as possible rather than isolated? [Are topics and situations interesting, enjoyable, and/or humorous? Is some thematic organization provided, such as through a story line | or episode? J a a a a Do tasks represent, of closely approximate, real-world tasks? Consid ticity may E Multiple-ch “Going 1 1. Amand Oyous Oar) Oyour Gwen: One! You Ols.a 3, Aman Oare: Oare: got 4, Gwen: wna Whe OWwhe 5. Aman or) Ogoir gon ‘Adapted frorCHAPTER 2 Principles ef Language Assessment 45 Consider the following two excerpts from tests, and the concept of authen- ticity may become a little clearer. Multiple-choice tasks—contextualized C88 2 Directions: After answering the questions, click the “Submit” button. “Going To” 1. Amanda; What ___ this weekend? ‘Q you are going to do ‘are you going to do | | Qyyour ona do | 2, Gwen: 'm not sure. anything spectal? ‘Ore you going to do. ‘© You are going to do (Os going to do, | | 3 Amanda: Melissa and! ___a party, Would you ike to come? | Oare going to are going | Ogoto | 4, Gwen: Pd love to!_? ‘What’ it gong to be ‘Who's going to be ‘Where’ it going to be | 5, Amands: i's ___to be at Ruth's house, Om ing © gonna. (sour) ‘Adapted from Sheila Viot, from Dave's ESL Café46 couereR 2. Principles of Language Assessment ‘Mulltiple-choice tasks—decontextualized “Going To” 1. What__ this summer? ‘A John is going to do, Bs John going to do, C. you're going to do anything special next weekend? ‘A.Rre you going to do B. You are going to do, G.ls going to do 3, She and 1_my English class tomorrow. Aare going to B. are going C. going to ‘The Giants are playing baseball on Wednesday. ‘A. What's it going to? BB. Who's it going to be? . Where's it going to be played? The ocean's. 1o.be at low tide later this morning. Ago 8.ong €. going to ‘The sequence of items in the contextualized tasks achieves a modicum of authen- ticity by contextualizing all the items in a story line. The conversation is one that ‘might occur in the real world, even if with a little less formality. The sequence of items in the decontextualized tasks takes the testtaker into five different topic areas with ‘no context for any, with the grammatical category as the only unifying element. Each sentence is likely to be written or spoken in the real world but only perhaps in five different contexts, Given the constraints of a multiple-choice format, on a measure of authenticity I would say the first excerpt is good and the second excerpt is only fair. 8. Does the test offer beneficial washback to the learner? ‘The design of an effective test should point the way to beneficial washback. A test that achieves content validity demonstrates relevance to the curriculum in question and thereby sets the stage for washback. When test items represent the various objec tives of a unit, and/or when sections of a test clearly focus on major topics of the unit, classroom tests can serve in a diagnostic capacity even if they aren't specifically labeled as such, Somet ination of t ical. Prepar is reviewiny In what I strategic ef task is tor ‘groups are By spe cover their tial by askit effort. The are likely to edge of the Some ‘enhance W: be an appr: can be part ‘ment has with a frest simply liste why. Journe ‘what they |Ciurrer 2 Principles of Language Assessment 47 The following checktist should help you to maximize beneficial washback in atest: }/, wasipack CHECKLIST 1, Is the test designed in such a way that you can give feedback that will be relevant to the objectives of the unit being tested? 2. Have you given students sufficient pretest opportunities to review the subject matter of the test? . In your written feedback to each student, do you include comments that will contribute to students’ formative development? 4, After returning tests, do you spend class time "going over" the test and offering advice on what students should focus on in the future? “After returning tests, do you encourage questions from students? 6. iftime and circumstances permit, do you offer students (especially the ‘weaker ones) a chance to discuss results in an office hour? a oo oO ON eee re I Sometimes evidence of washback may be only marginally visible from an exam- ination of the test itself. Here again, what happens before and after the test is crit- ical. Preparation time before the test can contribute to washback because the learner is reviewing and focusing in a potentially broader way on the objectives in question. In what I like to whimsically refer to as “wash forward,” students can be aided by strategic efforts to internalize the material being tested. An increasingly common ‘occurrence in student-centered classrooms is the formation of study groups whose task is to review the subject matter of an upcoming test. Sometimes those study groups are more valuable, in terms of measurable washback, than the test itself. By spending classroom time after the test reviewing the content, students dis- cover their areas of strength and weakness. Teachers can raise the washback poten- tial by asking students to use test results as a guide to setting goals for their future effort. The key is to play down the “Whew, I'm glad that’s over’ feeling that students are likely to have and play up the learning that can now take place from their know! ‘edge of the results. Some of the alternatives in assessment referred to in Chapter 1 may also ‘enhance washback from tests. (See also Chapter 6.) Selfassessment may sometimes be an appropriate way to challenge students to discover their own mistakes. This ‘can be particularly effective for writing performance: Once the pressure of assess- ‘ment has come and gone, students may be able to look back at their written work with a fresh eye. Peer discussion of the test results may also be an alternative to simply listening to the teacher tell everyone what they got right and wrong and ‘why. Journal writing may provide students a specific place to record their feelings, what they learned, and their resolutions for future effort. kok wk kk48 comeren2- Principles of Language Assessment ‘The five basic principles of language assessment have been expanded here into ‘eight essential questions you might ask yourself about an assessment. As you use the principles and guidelines to evaluate various forms of tests and procedures, be sure to allow each one of the five to take on greater or lesser importance, depending on the context. In large-scale standardized testing, for example, practicality is usually ‘more important than washback, but the reverse may be true of most classroom, tests, Validity is of course always the final arbiter. Remember, too, that these princi- ples, important as they are, are not the only considerations in evaluating or making, an effective test. Leave some space for other factors to enter in. ‘The next chapter focuses on how to design a test. These same five principles underlie test construction as well as test evaluation, along with some new concepts that expand your ability to apply principles to the practicalities of language assess ment in your own classroom. EXERCISES [Note: QD Individual work; (G) Group or pair work; (C) Whole-class discussion. } 1. (© Ask the class to volunteer brief descriptions of tests they have taken or given that illustrate, either positively or negatively, each of the five basic prin- ciples of language assessment that are defined and explained in this chapter. In the process, try to come up with examples of tests that illustrate (and dif- ferentiate) four kinds of reliability as well as the four types of evidence that support the validity of a test. 2. (U/C) Some assessment experts contend that face validity is not a legitimate form of validity because it relies solely on the perception of the test-taker rather than an external measure. Nevertheless, a number of educational assessment experts recognize the perception of the test-takers a very impor- tant factor in test design and administration. What is your opinion? How ‘would you reconcile the two views? 3. (G) In the section on washback, it is stated that "Washback enhances a number of basic principles of language acquisition: intrinsic motivation, autonomy, self-confidence, language ego, interlanguage, and strategic invest- ment, among others" (page 38). In a group, discuss the connection between wwashback and each of the above-named general principles of language learning and teaching. Describe specific examples or illustrations of each con- nection. If time permits, report your examples to the class. 4. (G) Ina small group, evaluate the assessment scenarios in the chart on pages 49-50 by ranking the six factors listed there from 1 t0 5 (with a score of 5 indicating that the principle is highly fulfilled and a score of 1 indicating very ow of no fulfillment). Evaluate the scenarios by using your best intuition in the absence of complete information for each context. Report your group's findings to the rest of the class and compare. 8 1. Standardized n proficiency tes production $ (Student) rece listing a total s for listening, gr and reading co 2. Timed improm| English (TWE® S receives a rep holistic score ra 3. One-on-one or overall oral pro S receives one between 0 and 4. gives a five-rr presentation in T (Teacher) eva rating sheet ind delivery, rappor rammar, and ¢ 5. S listens to a fit lecture and tak T makes indivic of Ss notes. 6. S writes a take- page essay on a TT reads paper at organization anc returns essay toSCENARIO. 1. Standardized multiple-choice Proficiency test, no oral or written produ 5 (Student) receives a report form listing a total score and subscores for listening, grammar, proofreading, and reading comprehension. 2. Timed impromptu test of written English (TWE® Test) 5 receives a report form listing one holistic score ranging between 0 and 6. 3. One-on-one oral interview to assess overall oral production ability S receives one holistic score ranging between 0 and 5. inute prepared oral class. presentation T (Teacher) evaluates by filling in a rating sheet indicating S's success in delivery, rapport, pronunciation, grammar, and content. 5. § listens to a fifteen-minute video lecture and takes notes. T makes individual comments on each of S's notes, 6. S writes a take-home (overnight) one- page essay on an assigned topic. Treads paper and comments on ‘organization and content only, then relums essay to $ for a subsequent draft Ciarree 2 Principles of Language Assessment 4950 cinrreR 2 Principles of Language Assessment SCENARIO 7. $ creates multiple drafts of a three- | page essay, peer- and T-reviewed, and turns in a final version. T comments on grammaticalrhetorical errors only and returns it to. 8, S assembles a portfolio of materials ‘over a semester-long course. T conferences with $ on the portfolio | at the end of the semester. | 5. (G) This chapter gives checklists to help you gauge and apply the five princi- ples of language assessment. In your group, talk about one principle. Describe a test that someone in your group took or gave. Discuss the fol: lowing question: Did it meet the criteria in the checklist? Report a summary of your discussion back to the class. 6. G) In our discussion of impact in this chapter, the suggestion was made that teachers can prepare students for tests by offering them strategies for preparing, taking, and benefiting from tests. These might be categorized as. “before, during, and after” strategies. “Before” strategies could include giving information about what to expect and suggestions for how to review. “During” strategies might involve tips for tackling items and time manage- ‘ment. “After” strategies, such as learning from one’s mistakes and setting future goals, could also benefit students. In a small group, design a checklist of test-taking strategies, perhaps with each group tackling just one of the three categories. Report your checklist back to the class. 7. (UG) In an accessible language class, ask the teacher to allow you to observe an assessment procedure that is about to take place (a test, an in-class peri- odic assessment, a quiz, etc.). Do the following: 1. Conduct a brief interview with the teacher before the procedure to get information on the purpose of the assessment and its place in the curriculum. . Observe (if possible) the actual administration of the assessment. cc. Arrange for a short interview with the teacher after the fact to ask any’ ‘questions you might have. Eval ciples of chapter. orally tot FOR YOUR Leung, C opmes secon Eribau This t issues itiser respor and re altern: practic Weir, Cyri Basing ‘This b in lang will er cially « ficult 1 classtc and teCourts 2 Principles of Language Assessment 51 Evaluate the effectiveness of the ssessment in terms of (a) the five basic prin- ciples of assessment and/or (b) the eight steps for test evaluation described in this, chapter. Present your findings either as a written report to your instructor and/or orally to the class, FOR YOUR FURTHER READING Leung, Constance. (2005). Classroom teacher assessment of second language devel- ‘opment: Construct as practice. In E. Hinkel (Ed.), Handbook of research in second language teacbing and learning (pp. 869-888). Mahwah, NJ: lawrence Eribaum Associates This highly informative ‘state-of-the-an” article summarizes a number of issues in classroom-based language assessment. For a relatively short article, itis exceptionally broad in its treatment of assumptions, claims, and caitic responses that have been faced in recent years, Constructs such as validity and reliability are examined and exemplified, and performance-based and alternative assessments are critically discussed, all with some down-to-earth practical examples that teachers can easily identify with. ‘Weir, Cyril J. (2005). Language testing and validation:An evidence-based approach, Basingstoke, England: Palgrave Macmillan. This book provides a wealth of information on research, issues, controver- sies, and solutions to problems that have arisen over the concept of validity in language testing. It offers teachers a theoretical and practical base that will enable them to evaluate their own classroom tests as well as commer- cially available tests. For the beginning student in the field, it could be di ficult reading, as it contains technical and complex discussion. However, classroom teachers will appreciate the many real-world examples of tests and test items.