Criticising dictionaries 173

14 Criticising dictionaries

Academic lexicography, or ‘metalexicography’, as pursued in university depart-

ments of English or Linguistics, is concerned not primarily with the compiling
of dictionaries – though academics may be involved in this, as consultants, for
example – but with researching and teaching about the whole business of
making dictionaries: their history, their typology, their structures, their users,
and so on (Hartmann 2001). One aspect of academic lexicography looks at the
products of commercial lexicography and subjects them to a rigorous critique,
usually resulting in a review; though academics are not the only ones who
review dictionaries. The process of critiquing and reviewing dictionaries is termed
‘dictionary criticism’.
One of the crucial issues for dictionary criticism is to establish a sound and
rigorous basis on which to conduct the criticism, together with a set of applica-
ble criteria. Hartmann (2001: 49) comments:

Anyone who has ever read (or written) a review of a particular dictionary
will know that generally agreed criteria and standards for the assessment of
quality and performance are still rare, if they can be said to exist at all.

This chapter discusses the business of dictionary criticism and proposes some ways
in which it may be undertaken and some guidelines for assessing dictionaries.

14.1 The business of criticism

In the brief section on dictionary criticism in his chapter on lexicography in
Solving Language Problems, Reinhard Hartmann (1996: 241) defines it as the
‘time-honoured’ activity of ‘evaluating and assessing lexicographic products’. It
is an activity that has a long history. Every new edition of a major dictionary
spawns reviews in all kinds of publication, from daily and weekly newspapers to
academic journals. But dictionary criticism is an activity, as Hartmann notes,
‘which has been beset by personal prejudice rather than noted for the applica-
tion of objective criteria’ (1996: 241). This concern is echoed in Noel Osselton’s
article – the only one with ‘Dictionary Criticism’ in its title – in the International
Encyclopedia of Lexicography (Hausmann et al. 1989). He notes ‘a surprising lack
174 Criticising dictionaries
of interest in general principles, with incidental sniping taking the place of any
real exploration’ (Osselton 1989: 229).
Dictionary reviews vary enormously in their approach and in their scope,
even those appearing in the same publication. Despite the fact that it is an
important means by which information about dictionaries is disseminated, little
attention has been paid to the methods and criteria underlying the business of
dictionary criticism. A ‘Note on Dictionary Criticism’ (by K.D.) from the Dic-
tionary Research Centre at the University of Exeter (date unknown) notes that:
‘Of general dictionaries there are a lot of reviews which lack validity and
reliability . . . what is needed is a wider discussion of the standards of assessment
of dictionaries.’ The note goes on to argue for a more objective evaluation of
dictionaries and greater clarity on what the criteria of assessment might be. It
concludes: ‘No definite theory of dictionary criticism has been established, and
it should be made more sophisticated as one field of lexicographical research.
Much remains to be done.’
Two main kinds of contribution have been made so far to the debate on
criteria for dictionary criticism. One has put forward proposals for guidelines or
criteria of reviewing. Roger Steiner’s ‘Guidelines for Reviewers of Bilingual
Dictionaries’ (Steiner 1984) would be a case in point, or Henri Béjoint’s seven
criteria for English monolingual learners’ dictionaries in his comparison of OALD,
COD and LDOCE (Béjoint 1978), or Robert Chapman’s four proposals for a
method of dictionary reviewing (Chapman 1977), or Herbert Ernst Wiegand’s
rather tongue-in-cheek ‘Ten Commandments for Dictionary Reviewers’
(Wiegand 1994).
The other kind of contribution, which is more recent, takes a set of diction-
ary reviews and subjects them to analysis, with the aim of discovering the
enduring concerns of dictionary reviewers. Jerzy Tomaszczyk (1988) took 120
reviews of general-purpose bilingual dictionaries and distilled the concerns of
the reviewers under the headings of: equivalents, directionality, reversibility,
alphabetisation, retrievability, redundancy, coverage, currency and reliability.
Martha Ripfel (1989) examined and compared the journalistic and academic
reviews of five German monolingual dictionaries, to identify the differing evalu-
ation and focus of the two types of review and the range of comments made
by each type. Günther Jehle (1990) looked at ‘popular’ and ‘academic’ reviews
of English and French monolingual learners’ dictionaries, with a focus on the
nature of a dictionary review as a text-type. He concludes his study thus (p. 300):
‘The practice of reviewing monolingual English and French learners’ dictionar-
ies in many cases unfortunately gives the impression that the reviewer has given
no prior thought to establishing the parameters within which his judgements
and critical assessments might have validity’ (my translation). We will attempt to
suggest some methodological principles and guidelines for dictionary criticism.

14.2 Method
Reviewing a dictionary is not like reviewing most other kinds of book publica-
tion. In the case of a normal book, the reviewer would expect to read the text
 Criticising dictionaries 175
in its entirety, perhaps some sections of text more than once. The reviewer of a
dictionary would not expect to read every word of the text: first of all, diction-
aries are not meant to be read like that, and usually dictionaries contain too
much text to make it a feasible undertaking – NODE, for example, claims 4
million words of text, while CED4 estimates that it contains 3.6 million words.
Also at issue is who might be an appropriate person to review a dictionary. In
general, reviewers – of books, plays, films, music – are chosen because they are
considered knowledgeable or expert in the subject matter or the techniques of
whatever it is they are reviewing. We should expect the reviewers of dictionar-
ies to be knowledgeable in lexicography. This is not always so, especially in
newspaper and magazine reviews: being a user of a dictionary appears to be
sufficient qualification sometimes, even though the same publication would not
think of asking just any reader to review a novel or a book of poetry.
A reviewer, then, must be knowledgeable about lexicography, a point made
by Roger Steiner in his contribution to Dolezal et al. (1994), and they also need
a sound methodology for critically reviewing a dictionary. As with any review-
ing, the first step must be to develop familiarity with the work that is being
assessed. With dictionaries, this means, first of all, reading the often neglected
front-matter: the preface, the guide to using the dictionary, the list of staff and
consultants, and so on. This will usually give a preliminary view of the scope of
the dictionary, its intended users, and the types of lexical (and other) informa-
tion that are claimed to be included. Familiarisation will also include browsing
the main body of the dictionary and reading a variety of types of entry, as a
means of gaining an impression of the flavour of the particular dictionary under
review. Finally, some dictionaries have back-matter (appendices), which may
contain gratuitous additional information (e.g. counties of the UK and states of
the USA) or provide useful lexical information (e.g. affixes and combining
forms).
For a detailed assessment of the content of a dictionary, Robert Chapman
(1977) suggests that random sampling of entries should be used, such as ‘the
tenth main entry on every twentieth page’, in order to yield a manageable set of
entries (e.g. 50); and he proposes that each of these entries should be scrutinised
carefully for ‘accuracy, completeness, clearness, simplicity, and modernity’ (cri-
teria from McMillan 1949). Random sampling of entries ensures that a reviewer
does not look for a predetermined list of favourite items. However, a reviewer
needs to make sure that the resulting sample is representative, so that it contains
at least one member of each word class, that polysemy is well represented, and
that there is a spread across general and specialist lexemes. A random sample
may need to be supplemented by the reviewer’s checklist. Likewise, where a
dictionary has special additional features (e.g. synonym essays (LDEL2) or usage
notes (NODE, ECED, etc) ), these too need to be taken into account.
One of Chapman’s other suggestions is that the reviewing of dictionaries
should be undertaken by a team of reviewers. His conception of this seems to be
that each reviewer would be a specialist in some area of vocabulary and would
contribute an assessment of the treatment of the ‘definitions in their own fields’
(Chapman 1977: 158). Where team reviewing has been undertaken more recently
176 Criticising dictionaries
(e.g. the well organised and comprehensive Japanese reviews of COD8 (Higashi
et al. 1992) and of LDEL2 (Masuda et al. 1994) ), each member of the team has
taken a different aspect of linguistic description (pronunciation, definition, usage,
etymology, etc.) rather than vocabulary specialism, which is probably a more
sensible division of labour. Team reviews allow a more thorough treatment of
each aspect of a dictionary’s lexical description, both by enabling more extensive
sampling to be undertaken and by tapping into a reviewer’s specialist interest.

14.3 Criteria: internal and external

Criteria for evaluating a dictionary can be derived from two possible sources:
‘internal’ criteria and ‘external’ criteria. Internal criteria derive from what a
dictionary says about itself, or what the editors claim for the dictionary. External
criteria derive from metalexicography (the academic study of dictionaries), tak-
ing into account the linguistic requirements for a lexical description, as well as
considerations of dictionary design and production.
All dictionaries, in their preface or their blurb, make claims about their fea-
tures that distinguish the current edition from previous ones, or demonstrate
the superiority of this dictionary over its rivals. A reviewer can take these claims
as a basis for reviewing, to investigate whether the claims are borne out in the
dictionary’s practice: these are the internal criteria. For example, NODE con-
tains an extensive ‘Introduction’, whose principal aim is to ‘explain some of the
thinking behind these new approaches’ (p. ix). The Introduction makes the
following statement:

each word has at least one core meaning, to which a number of subsenses
may be attached . . . Core meanings represent typical, central uses of the
word in question in modern standard English . . . The core meaning is the
one that represents the most literal sense that the word has in ordinary
modern usage.
(p. ix)

These are testable statements. And they are tested by Sidney Landau in his
review of NODE (Landau 1999); he comes to the conclusion that ‘NODE’s
defining strategy is ambitious, and it does not always succeed . . . Nevertheless,
in most cases, the defining strategy does work.’ (p. 252)
Many shorter reviews take a dictionary’s view of itself at least as a starting
point for the evaluation, if not as the basis for the whole approach. Newspaper
and magazine reviews routinely depend on what the editors or publisher say
about their product, often with little attempt to test the sometimes exaggerated
claims. In that sense, there is a danger that a critique that relies solely on internal
criteria may be biased too much in favour of the dictionary, unless a radically
critical stance is taken to the claims that are made.
In using external criteria, a reviewer begins from a different standpoint. The
criteria are determined prior to the review; they arise from the accumulated
insights of the academic community (e.g. Hudson 1988; Ilson 1991). This does
 Criticising dictionaries 177
not mean that they need be overly ‘academic’, since the community must take
account of the fact that dictionaries are as much reference works aimed at par-
ticular groups of users as they are linguistic descriptions of the lexical resources
of the language. Rundell (1998: 316) suggests two criteria for the evaluation of
improvements in the development of learners’ dictionaries:

the description of a language that a dictionary provides corresponds more

closely to reliable empirical evidence regarding the way in which that lan-
guage is actually used; the presentation of this description corresponds more
closely to what we know about the reference needs and reference skills of
the target user.

There is a need, therefore, for two sets of external criteria for the evaluation of
dictionaries: one set relates to the reference function of dictionaries and the
user’s perspective, and is largely about presentation and accessibility; the other
relates to the recording function of dictionaries, and is largely about content.
Presentation and content overlap and interact with each other (e.g. the core
sense and subsense division in NODE), so this is to some extent a false dicho-
tomy. However, it will provide a useful framework for establishing criteria for
dictionary reviewing.

14.4 Presentation
How a dictionary presents its material has an important influence on the acces-
sibility of the information for its target users. In the preface to the first edition of
the COD (1911), the Fowler brothers commented that they had used ‘the
severest economy of expression – amounting to the adoption of telegraphese –
that readers can be expected to put up with’ (p. iv). How they determined
readers’ tolerance levels is not stated, but they are surely rather different from
those of today’s dictionary users. Aspects of presentation that should be consid-
ered by a dictionary reviewer include at least the following.

Page layout
The size of the page varies with the size of the dictionary (desk, concise, pocket
– see 3.2). It is usual to have two columns, though three may be found in some
dictionaries (e.g. NODE, ECED). A significant feature is the amount of white
space, determined by the size of the margins, the spacing between entries, the
inclusion of other material to break up the text (e.g. usage notes, diagrams,
illustrations). All these can have an effect on the appearance of the dictionary
page. Attractive page layout improves accessibility.

Layout of the entries

Traditionally dictionaries have tended to pack all the information for a head-
word within a single paragraph, to save space; more recent editions have begun
178 Criticising dictionaries
to unpack the paragraph and to use a new line to begin a new set of information.
For example, NODE begins a new line for: a different word class, derivatives,
phrases, etymology. In some learners’ dictionaries (LDOCE3, CIDE), the lay-
out has been used to enhance the user’s access to specific meanings of polysemous
words.

Length of entries
This is determined largely by the practice of ‘nesting’, where derivatives, com-
pounds, idioms and so on are included within a single entry under a root word.
Chambers uses nesting extensively, as does the COD up to the seventh edition
(1982). The use of separate headwords for compounds and derivatives, where
these are individually defined, creates shorter and more numerous entries, and
more white space, so enhancing page layout. Likewise the inclusion of abbre-
viations, affixes and combining forms as headwords (rather than in appendices)
creates more and shorter entries and aids accessibility.

Abbreviation
The use of abbreviations, like nesting, saves space, an important consideration
in dictionary making. The Fowlers noted this as a feature of COD1; their
assumption was that the users of the COD would understand and cope with a
high degree of abbreviation. That is no longer the case, and many recent dic-
tionaries (e.g. CED, NODE) now include the full forms of items such as word
class labels and names of languages in etymologies. Fewer abbreviations mean
greater accessibility.
Academic reviewers tend to concentrate on the content of dictionaries, but
presentation and accessibility should not be ignored, because they make a sig-
nificant contribution to enabling users to be successful in extracting information
from the dictionary.

14.5 Content
Hudson’s (1988: 310–12) ‘checklist of types of lexical fact’ would provide a
starting point for criteria of content, or alternatively Ilson’s (1991) more induc-
tively arrived at set of headings would serve. Such a set of criteria would en-
compass at least the following.

Range of vocabulary
Modern dictionaries are keen to claim that they have included the latest words
from areas where neologisms are common (e.g. business, information technol-
ogy, the environment, medicine), as well as coverage of other national varieties
of English (e.g. American, Australian). A reviewer would need to determine
whether, for its size and scope, the dictionary had adequate coverage of up-
 Criticising dictionaries 179
to-date, technical, international and, if appropriate, regional lexis. Where a
dictionary includes such items, geographical and biographical entries would also
come under this heading.

Word formation
On the one hand, this relates to whether affixes and combining forms are treated
as headwords or gathered in an appendix, or not given any attention at all (cf.
Pr[ic 1999). On the other hand, it relates to the treatment of derived and com-
pound words, what the criteria are for separate headword status as against nested
run-on. Also relevant here is the treatment of noun/adjective pairs that are not
cognate, e.g. church – ecclesiastical, law – legal, mind – mental, lung – pulmonary. The
judgement to be made is whether the account of word formation enables a user
to ascertain the formal (morphological) relations between words.

Homographs
The usual basis for more than one headword for a single spelling is different
etymologies. In some dictionaries (e.g. LDEL) each word class that a lexeme
belongs to occasions a new headword. In COBUILD1 there is only one head-
word per spelling, whereas more recent monolingual learners’ dictionaries
(LDOCE3, CIDE) have multiple entries based on meaning. The criteria for
determining what is a headword have important consequences for lexical de-
scription as well as for accessibility.

Sense division
For words that have multiple meanings (senses), dictionaries do not always make
clear how the senses have been established (cf. Allen (1999) on ‘lumpers and
splitters’), or the order in which they have been arranged. A tendency is emerg-
ing (e.g. NODE, COD10) to pull back from the over-differentiation of senses
that has occurred in the past (CED, Chambers). The issue here relates both to the
adequacy of the lexical description and to how straightforward it is for the user
to find the desired sense.

Defining
This is usually seen as the crucial task of the lexicographer, and there are some
well established defining styles, notably the analytical (Kipfer 1984: 66–8).
However, these have been extended in recent years, especially in learners’ dic-
tionaries (e.g. with whole sentence definitions). Some dictionaries include a
certain amount of encyclopedic information in their definitions, especially for
words referring to flora and fauna. Not only does a reviewer need to assess the
adequacy of the definitions, but also whether they are stylistically appropriate
for the intended users.
180 Criticising dictionaries
Beyond denotation
Under this heading is included other aspects of a word’s lexical behaviour over
and above its denotation, such as its lexical relations (synonymy, antonymy,
hyponymy – as indicated systematically in COBUILD1), its typical colloca-
tions, and any shared connotations. Sometimes this information is incorporated
into definitions, but it is rarely handled systematically, though the ‘synonym
essays’ in LDEL2 and ECED constitute a rare exception. Yet this information
also contributes to an understanding of a word’s meaning.

Pronunciation
There are two issues here: the transcription system, which is almost universally
IPA now in British dictionaries, and the accent to be represented. While many
dictionaries now give alternative American pronunciations, no account is taken
of the fact, for example, that the majority of British speakers say /bgt/ rather
than /bkt/ and /græs/ rather than /grp:s/. Some native speaker dictionaries
(e.g. NODE) are now giving pronunciations only for words that are problem-
atical, but what may be designated a problematical pronunciation is a matter of
judgement (Allen 2000).

Grammar
Dictionaries have traditionally given word class (part-of-speech) labels, and for
verbs distinguished ‘transitive’ and ‘intransitive’ uses; NODE acknowledges that
these may be opaque terms for modern users and substitutes ‘with obj’ and ‘no
obj’. Rarely have native speaker dictionaries given much more information
about grammar beyond this; CED, and now NODE, are exceptions. By con-
trast learners’ dictionaries have aimed for full coverage, which raises the ques-
tion about how grammatical information is represented for effective access.
A reviewer needs to evaluate how much information about the grammatical
operation of words is necessary for a dictionary to fulfil its recording function,
as against the need not to provide too much unnecessary information for
the intended users.

Usage
Dictionaries routinely label words or senses of words with ‘restrictive’ labels, to
indicate that the word or sense may be used only in a specific context. Such
labels may relate to: time (obsolete, archaic), dialect (North American, Austral-
ian English, Scottish), formality (informal, colloquial), evaluation (derogatory,
pejorative, euphemistic), status (slang, taboo), field or topic (Astronomy,
Music, Telecommunications). The extent to which dictionaries are consistent
in using their range of usage labels and how they apply them are matters for
the critic to evaluate. Some dictionaries include ‘usage notes’, especially to give
 Criticising dictionaries 181
guidance on controversial areas, e.g. the preposition to be used after different, the
distinction between disinterested and uninterested. A critic may note how ‘con-
servative’ or ‘progressive’ a stance is taken by a dictionary on such issues.

Examples
All dictionaries give example sentences or phrases to illustrate word meaning,
grammar or usage. They are particularly numerous and prominent in learners’
dictionaries, where they are seen as playing a crucial role. A number of ques-
tions need to be asked by the critic, relating to: the extent of the use of exam-
ples, what role they are seen to play in exemplification, where the examples
come from (corpus or invented), and how consistently the dictionary’s policy
on examples is implemented.

Etymology
Since the etymological dictionaries of the eighteenth century, it has been cus-
tomary to include information about etymology in native speaker dictionaries,
though not in learners’ dictionaries (Chapter 10). It might be argued that such
information has no place in a dictionary of the contemporary language and
should be confined to ‘historical’ dictionaries (such as OED and SOED), though
Hudson (1988) includes etymology in his checklist of lexical facts. The amount
of etymological detail that general dictionaries include is variable; in some cases
it is just the language of immediate origin, or the etymology may be traced back
as far as possible, and perhaps with cognates in related languages. It is a matter of
critical evaluation whether the information in the dictionary under review is
appropriate to its size, purpose and intended users.

Special features
A dictionary will often seek to distinguish itself from its rivals by including a
special feature, e.g. the synonym essays and other boxed comments in LDEL2,
usage and other notes in NODE, word formation boxes in COD10, misspell-
ings in ECED, frequency information in COBUILD and LDOCE3. Some-
times these are a genuine enhancement of the information that the dictionary
gives; sometimes they are more of a marketing gimmick. Other special features
may be incorporated in the front-matter or the appendices, e.g. the essay on
English as a world language in CED, or the punctuation guide in COD8/9.
The question is whether they add to the lexical description and the coverage
and usefulness of the dictionary.

14.6 Perspective
Besides following an appropriate method and making judgements against an
explicit set of criteria, it is also possible for a reviewer to conduct dictionary
182 Criticising dictionaries
criticism from a number of different perspectives. The discussion so far has
more or less presupposed that dictionary criticism is undertaken from the per-
spective of the academic linguist or metalexicographer, applying lexicological
and lexicographical theory and insight to the task. Certainly, in the reviews that
have appeared in the International Journal of Lexicography, this has been, under-
standably, the predominant perspective adopted. But it has not always been
appropriate. For example, there was a review of OALD4 by Dwight Bolinger
(1990), which took the dictionary to task for not representing accurately some
very subtle syntactic peculiarities of verbs, which, had they been so treated,
would have probably baffled most users of the dictionary. Bolinger had, un-
characteristically, not appreciated the need for what Rundell has called ‘a more
utilitarian lexicography’ (Rundell 1998: 337), where there is a tradeoff between
the needs of the user and the meticulous accuracy of the lexical description.
An alternative perspective to that of the metalexicographer would be that, for
example, of the target user. The dictionary would be judged, by the method
and on the criteria stated, from the point of view of the needs, expectations,
prior knowledge and reference skills of the intended group of users. For an
example, see Jackson (1995) which makes a comparison of LDOCE2 and the
German learners’ dictionary, Langenscheidts Großwörterbuch Deutsch als Fremdsprache.
A third possible perspective would be that of the language teacher – particu-
larly in respect of learners’ dictionaries – who would judge the dictionary from
the point of view of its suitability for the language teacher’s task and for the
students that they are teaching, whether in a first or a second language context.
Higashi et al. (1992) review COD8 from this perspective in the Japanese con-
text, perhaps inappropriately, since COD8 was not intended as a pedagogical
dictionary, although the COD seems to have been used widely for this purpose
in Japan.
The reviewer’s perspective can act as a focus for the attention to be paid in
the critique to different aspects of the dictionary’s presentation and content,
especially since it is difficult to treat every aspect in a single review.

14.7 Purpose
In conclusion, we may reflect on the purposes for which dictionary criticism is
carried out. Some reviews, especially in newspapers and magazines, have as
their main purpose to inform the public of the existence of a new edition of,
usually, a well-known dictionary; the content of the review then often reflects
the publisher’s press release or the dictionary’s blurb. Other dictionary reviews,
like any book review, are directed at an interested public (teachers, students,
crossword addicts) and have as their purpose to inform this audience of the
content of the dictionary and its fitness for their needs.
Reviews of dictionaries that appear in academic journals, such as the Inter-
national Journal of Lexicography, while informing the journal readers about the
existence and contents of the dictionary, are also intended in many instances to
make a contribution to academic lexicography. Such reviews are often more
 Criticising dictionaries 183
thorough, pursue a more rigorous methodology, and draw on the accumulated
wisdom and expertise of the academic community of dictionary scholars. There
is one further purpose that academic reviews may have. Since their critique is
drawn from an expert knowledge of dictionaries, dictionary making and dic-
tionary use, they often propose ways in which dictionaries may be improved.
So, they are offering advice to working lexicographers and dictionary publish-
ers, and contribute towards the development of both practical and academic
lexicography.
If dictionary reviews are to fulfil this function, or indeed if they are to
make a serious contribution to the academic study of lexicography, then it is
important that dictionary criticism is conducted on a sound basis, with a clear
methodology and a set of explicit criteria.
One of the purposes of this book, and of this chapter in particular, is to give
you, the reader, the background to enable you to look at dictionaries, both
historical and contemporary, with a more informed insight. On the basis of
your study, you might attempt the review of a dictionary.

14.8 Further reading

There is no full-length treatment of dictionary criticism. The place to start is
with Reinhard Hartmann’s Teaching and Researching Lexicography (2001), where
he deals with the topic in Chapter 4, Sections 4.3 and 4.4, which also contain
references to other relevant articles and books.
From there it would be useful to read some of the reviews that have
appeared, for example, in the International Journal of Lexicography. The reviews,
mentioned earlier, of COD8 by Higashi et al. (1992) and of LDEL2 by Masuda
et al. (1994), are particularly recommended; but most numbers of the journal
contain dictionary reviews of varying extent and comprehensiveness. The other
journal in which dictionaries are regularly reviewed is English Today.