EVALUATING MATERIALS

CONTENTS:
➢What is Materials Evaluation

➢Evaluators Theory of Learning and Teaching

➢Second Language Acquisition (SLA)

➢Types of Materials Evaluation

➢Developing Criteria for Materials Evaluation

What is Materials Evaluation?

➢ Materials evaluation may be defined as a

procedure or a systematic appraisal
measuring the potential value(s) of materials
on learners in relation to their
objectives (Tomlinson 1998, 2003).
- is a procedure that involves measuring the value (or potential value) of
a set of learning materials. It involves making judgments about the effect
of the materials of the people using them and it tries to measure some or
all of the following;

• the appeal of the materials to the learners;

• the credibility of the materials to learners,
teachers, and administrators;
• the validity of the materials (i.e., Is what they
teach worth teaching?);
• the ability to the materials to motivate learners;
• The value of the materials in terms of short-term learning
important (for example, for performance on tests and
examinations)
• the value of the materials in terms of long-term learning
(of both language and of communication skills)
• the assistance given to the teachers in term of
preparation, delivery, and assement;
• the flexibility of the materials (e.g., the extent to which it is
easy for a teacher to adapt the materials to suit a
particular context);
• the match with administrative requirements (e.g.,
standardization across classes, coverage of a syllabus,
preparation for an examination).
 The Evaluators Theory of
Learning and Teaching

All teacher develop theories of learning and teaching which

they apply in their classrooms. Many researchers (ex. Schon, 1983)
argue that it is useful for teachers to try to achieve an articulation
of their theories by reflecting on their practice. For example, Edge
and Wharton (1998, p. 297 argue that reflective practice can not
only lead to perceived improvements in practice but, more
importantly, to deeper understanding of the area investigated.
Here are some theories of Brian
Tomlinson and other teachers practice:
• Language learning succeed best if learning is a positive,
relaxed and enjoyable experience.
• Language teachers tend to teach most successfully if
they enjoy their role and if they can gain some
enjoyment themselves from the materials they are using.
• Learning materials lose credibility for learners if they
suspect that the teacher does not value them.
• Each learner is different from all the others in a class
in terms of his or he personality, motivation, attitude,
aptitude, prior experience, interests, needs, wants
and preferred learning style.
• Each learner varies from day to day in terms of
motivation, attitude, mood, perceived needs and
wants, enthusiasm and energy.
Second Language Acquisition SLA)

SLA research is so far inconclusive and has stimulated

many disagreements and debates (e.g., about the value of
the explicit teaching of discrete language points).However,
there is now a sufficient consensus of opinion from certain
facilitating features of language learning for them to be
useful in helping to articulate the principles to be used as a
basis of material evaluation .
Tomlinson discussed the principles of second language acquisition
that relevant to the development of materials for the teaching of
languages. Some of these principles are summarized below:
➢ Materials should achieve impact (through novelty, variety, surprise,
bizarreness, attractive presentation and appealing content).
➢ Materials should help learners to feel at ease (ex. through the use of white
space to prevent clutter and the use of texts and illustrations which they
can relate to their own culture, through a supportive approach which is not
always testing them and through the use of a personal voice).
➢ Materials should help the learners to develop confidence (ex. through
'pushing' learners slightly beyond their existing proficiency by involving
them in tasks which are challenging but achievable).
➢ Materials should take into account that learners differ in learning styles
(Oxford and Anderson, 1995; Oxford, 2002; Anderson, 2005) (and
should therefore ensure that they cater for learners who are
predominantly visual, auditory, Kinesthetic, studial, experiential, analytic,
global, dependent or independent.
➢ Materials should take into account that learners differ in affective
attitudes (Wenden and Rubin, 1987) (and therefore materials should
offer variety and choice).
➢ Materials should provide opportunities for outcome feedback (i.e.,
feedback on the effectiveness of the learner in achieving communication
objectives rather than just feedback on the accuracy of the output).
Richards (2001,p. 264) suggests a rather different and briefer
list of the qualities each unit in the materials should reflect:

• Gives learners something they can take away from the lesson;
• Teaches something learners feel they can use;
• Gives learners a sense of achievement;
• Practices learning items in an interesting and novel way;
• Provides a pleasurable learning experience;
• Provides opportunities for individual practice;
• Provide opportunities for personalization;
• Provide opportunities for self-assessment of learning.
Types of Materials
Evaluation
Pre-use Evaluation

Pre-use evaluation involves making predictions about the

potential value of materials for their uses. It can be context-
free, as in a review of materials for a journal, context-
influenced as in a review of draft materials for a publisher
with target users in mind or contexts dependent, as when a
teacher selects a coursebook for use with her particular
class.
Whilst-use Evaluation
➢ This involves measuring the value of materials while
using them or while observing them being used. It can
be more objective and reliable than pre-use
evaluation as it makes use of measurement rather than
prediction.
➢ It can measure short-term memory through observing
learner performance on exercises but it cannot
measure durable and effective learning because of
the delayed effect of instruction.
Post-use Evaluation
Post-use evaluation is probably the most valuable (but least
administered) type of evaluation as it can measure the actual
effects of the materials on the users. It can measure the short-
term effect as regards motivation, impact, achievability, instant
learning, etc., and it can measure the long term effect as
regards durable learning and application. It can answer such
important questions as:

• What do the learners know which they did not know before
starting to the materials?
• What do the learners still not know despite using the
materials?
• What can the learners do which they could not do
before starting to use the materials?
• What can the learners still not do despite using the
materials?
• Did the teachers find the materials easy to use?
• Did the materials help the teachers to cover the
syllabus?
Developing Criteria for
Materials Evaluation
1. Brainstorm of list of
universal Criteria.
Universal criteria are those which would apply to any language
learning materials anywhere for any learners. For example, they
would apply equally to a video course for 10-years-old in
Argentina and an English for academic purposes textbook for
undergraduates in Thailand. Brainstorming a random list of such
criteria (ideally with other colleagues) is a very useful way of
beginning an evaluation.
 Examples of universal criteria
would be:
• Do the materials provide useful opportunities for the
learners to think for themselves?
• Are the target learners likely to be able to follow the
instructions?
• Are the materials likely to cater for different preferred
learning styles?
• Are the materials likely to achieve affective engagement?
2. Subdivide some of the
Criteria
If the evaluation is going to be used as a basis for
revision or adaptation of the materials, or if it is going to be
a formal evaluation and is going to inform important
decisions. It is useful to subdivide some of the criteria into
more specific questions.
For example:
Are the instructions:

• succinct? • separated?
• sufficient? • sequenced?
• self-standing? • staged?
• standardized?
3. Monitor and revise the
list of universal criteria
Monitor the list and rewrite it according to the following criteria;
Is each question an evaluation question?
If a question is an analysis question (ex. ‘Does each unit
include a test?’) then you can only give the answer 1 or 5 on
the 5-point scale which is recommended later in this suggested
procedure. However, it it is an evaluation question (ex. ‘To
what extent are the tests likely to provide useful learning
experiences?’) then it can be graded at any point on the scale.
Does each question only ask one
question?
Many criteria in published lists ask two or more questions and
therefore cannot be used in any numerical grading of the
materials. For example, Grant (1987) includes the following
question which could be answered ‘Yes; No’ or ‘No; Yes’: Given
the average age or the students, would they using it? (p. 122.)
This question could be usually rewritten as:

1. Is the book likely to be attractive to your students?

2. Is it suitable for that age of the students?
3. Are the students likely to enjoy using it?
Is each question answerable?
This might seem an obvious but in many published lists of
criteria some questions are so large and so vague that they
cannot usually be answered.

For example:
• Is it culturally acceptable? (Grant, 1987, p, 122)
• Does it achieve an acceptable balance between
knowledge about the language and practice in using the
language? (Ibid)
Is each question free of dogma?
The question should reflect the evaluators principles of
language learning but should not impose a rigid
methodology as a requirement of the materials. If they do,
the materials could be dismissed without a proper
appreciation of their potential value. For example, the
following examples make assumptions about the
pedagogical procedures of course book which not all
course book actually follow:
• Are the various stages in a teaching unit ( what
you would probably call presentation, practice
and production) adequately developed?
(Mariani, 1983, p. 29).
• Does the sentences gradually increase in
complexity to suit the growing reading ability of
the students? (Daoud and Celce-Murcia, 1979,
p. 304).
4. Categorize the List
It is very useful to rearrange the random list of universal criteria into
categories which facilitate focus and enable generalizations to be made. An
extra advantage of doing this is that you often think of other criteria related
to the category as you are doing the categorization exercise.
Possible categories for universal criteria would be:

• Learning Principles • Activities

• Cultural Perspective • Methodology
• Topic content • Instructions
• Teaching Points • Design and
• Texts Layout
5. Develop media-specific
criteria
These are criteria which ask questions of particular
relevance to the medium used by the materials being evaluated
(e.g. criteria of books, for audio cassettes, for videos, etc.
Examples of such criteria would be:

• Is it clear which sections the visuals refer to?

• Is the sequence of activities clearly signalled?
• Are the different voices easily distinguished?
• Do the gestures of the actors help to make the language
meaningful in realistic ways?
6. Develop content-specific
criteria
These are criteria which relate to the topics and/or teaching
points of the materials being evaluated. Thus there would be a
set of topic related criteria which would be relevant to the
evaluation of a business English textbook but not to a general
English course book; and there would be a set of criteria
relevant to a reading skill Book which would not be relevant to
the evaluation of a grammar practice book and vice versa.
(Tomlinson, 1999, p. 11).
Example of content-specific criteria would be:

• Do the examples of business texts (e.g. letter,

invoice, etc.) replicate features of real-life
business practice?
• Do the reading texts represent a wide and
typical sample of genres?
7. Develop age-specific
criteria
These are criteria which relate to the age of the
target learners. Thus there would be criteria which are
only suitable for 5 years old, 10 years old, for
teenagers, for young adults and for the mature adults.
These criteria would relate to cognitive and affective
development, to previous experience, to interests and
to wants and needs.
Examples of age-specific criteria would be:

• Are there short, varied activities which are likely to

match the attention span of the learners?
• Is the content likely to provide and achievable
challenge in relation to the maturity level of the
learners?
8. Develop Local criteria
These are criteria which relate to the actual or potential
environment of use. They are questions which are not concerned
with establishing the value of the materials but rather with
measuring the value of the materials for particular learners in
particular circumstances. It is set of criteria which is unique to the
specific evaluation being undertaken and which is ultimately
responsible for most of the decisions made in relation to the
adoption, revision or adaptation of the materials.
 Typical feature of the environment which
would determine this set of materials are:

• The types of institutions • The objectives of the courses

• The resources of institutions • The intensity and extent of the
• The background, needs and wants teaching time available
of the learners • The target examinations
• The language policies in operation • The amount of exposure to the target
• The syllabus language outside the classroom
 Examples of local criteria would be:

• To what extent are the stories likely ton interest 15 years

old boys in Turkey?
• To what extent are the reading activities likely to prepare
the students for the reading questions in the Primary School
Leaving Examination in Singapore?
• To what extent are the topics likely to be acceptable to
parents of students in Iran?
 9. Develop other criteria
Which it might be appropriate to develop could include teacher-
specific, administrator-specific, gender-specific, culture-specific or L1-
specific criteria, and especially in the case of a review for a journal,
criteria assessing the match between the materials and the claims
made by the publisher for them

10. Trial the criteria

It is important to trial the criteria (even prior to a small, fairly
informal evaluation) to ensure that the criteria are sufficient,
answerable, reliable, and useful. Revision can be made before the
actual evaluating begins.
11. Conducting the Evaluation
These are the most effective way of conducting an
evaluation:
• Make sure that there is more than on evaluator;
• Discuss the criteria to make sure there is equivalence of
interpretation;
• Focus in a large evaluation on a typical unit for each level
(and then check its typicality by reference to other units);
• Write comments at the end of each category;
• Record the comments shared by the evaluators;
Conclusion

➢Materials evaluation in initially a time consuming

and difficult undertaking.
➢Approaching it in the principled systematic and
vigorous ways suggested above cannot the
materials being evaluated but can also help the
evaluators to learn a lot about materials above
learning and teaching about themselves
END