EDUCATING PHYSICIANS

R E S E A R C H R E P O R T

Reasons Reviewers Reject and Accept Manuscripts:

The Strengths and Weaknesses in Medical
Education Reports
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 01/12/2022

Georges Bordage, MD, PhD

ABSTRACT

Purpose. Scientific journals rely on peer review to (SD = 5.7) reasons. The top ten reasons for rejection
maintain the high quality and standards of papers ac- were: inappropriate or incomplete statistics; overinterpre-
cepted for publication. The purpose of this study was to tation of results; inappropriate or suboptimal instrumen-
explore the strengths and weaknesses of medical educa- tation; sample too small or biased; text difficult to follow;
tion reports by analyzing the ratings and written com- insufficient problem statement; inaccurate or inconsistent
ments given by external reviewers. data reported; incomplete, inaccurate, or outdated review
Method. The author conducted a content analysis of re- of the literature; insufficient data presented; and defective
viewers’ comments on 151 research manuscripts submit- tables or figures. The main strengths noted in accepted
ted to the 1997 and 1998 Research in Medical Education manuscripts were the importance or timeliness of the
conference proceedings. The negative comments on 123 problem studied, excellence of writing, and soundness of
manuscripts that received ‘‘questionable, probably ex- study design.
clude’’ or ‘‘definitely exclude’’ overall ratings from at least Conclusion. While overstating the results and applying
one reviewer were evaluated. A similar analysis was per- the wrong statistics can be fixed, other problems that the
formed on reviewers’ positive comments for 28 manu- reviewers identified (ignoring the literature, designing
scripts recommended unanimously for acceptance. poor studies, choosing inappropriate instruments, and
Results. On average, four peers (4.1, SD = 0.97, range writing poor manuscripts) are likely to be fatal flaws war-
= 2–6) reviewed each manuscript. Of those recom- ranting rejection.
mended for exclusion, a mean of 2.3 reviewers recom- Acad. Med. 2001;76:889–896.
mended exclusion and each reviewer wrote a mean of 8.1

Journal editors rely on peer review to high-quality research in their fields. Re- First International Congress on Peer
maintain high quality and standards in viewers typically assess the quality of Review in Biomedical Publishing.2
the papers they accept for publication, manuscripts according to two main cri- However, few studies exist that analyze
and researchers and educators rely on teria: ‘‘contribution to the field’’ and the the content of reviewers’ comments
peer-reviewed journals as sources of ‘‘adequacy of the research design.’’ 1 when reviewers are recommending re-
There is a growing body of research on jection or acceptance of a manuscript.
journal peer review. For example, Gilbert and Chubin1, p. 108 conducted
Dr. Bordage is professor and director of graduate JAMA has dedicated three complete is- such an analysis for a sample of review-
studies, Department of Medical Education, College
of Medicine, University of Illinois at Chicago.
sues in the past decade (March 9, 1990; ers’ comments on manuscripts that had
July 13, 1994; and July 15, 1998) to been rejected from Social Studies of Sci-
Correspondence and requests for reprints should be
addressed to Dr. Bordage, Department of Medical peer review studies and essays, and the ence, an interdisciplinary specialty jour-
Education, College of Medicine, University of Illinois Council of Biology Editors (now the nal. They found that the most frequent
at Chicago, 808 South Wood, 986 CME, Chicago,
IL 60612-7309; telephone: (312)996-7349; fax: Council of Science Editors) also pub- reason reviewers offered for rejecting a
(312)413-2048; e-mail: 具[email protected]典. lished a book of papers in 1991 from the manuscript was ‘‘poor argumentation,’’

ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 889

that is, failing to make a convincing form to evaluate each manuscript in for acceptance (the positive com-
case. Other reasons frequently given in- eight areas: Problem Statement and ments). For manuscripts that were rec-
cluded poor writing, ignorance of the Background, Research Design, Sam- ommended for rejection, only the com-
literature, lack of novelty, and misun- pling, Instrumentation and Data Col- ments of reviewers recommending
derstanding or misapplying the data or lection, Results, Conclusion, Writing, rejection were analyzed.
the literature. While reviewers do write and Importance. Each area is rated on The analysis was performed solely by
comments, the level of agreement a five-point scale (excellent, good, fair, the author. To avoid categorization bias,
among reviewers remains highly varia- unsatisfactory, and not applicable). The the ratings and comments were ana-
ble. Assessing studies from different reviewers are also asked to provide a lyzed in a staggered fashion according to
areas of science, Chubin and Hackett global rating using a four-point scale years (i.e., 20 from 1997 followed by 20
reported poor agreement among review- (definitely include; acceptable, probably from 1998, etc.). Reasons (comments)
ers, with inter-rater reliability in the include; questionable, probably exclude; were tallied only once per reviewer.
0.25 range.1 The research confirms what definitely exclude) and to write detailed Whenever a comment (e.g., ‘‘question-
editors have known: Reviewers for any comments on the merits or shortcom- able randomization’’) could belong to
given manuscript focus on different is- ings of the manuscript. Historically, the more than one category of reason (e.g.,
sues. acceptance rate for RIME papers is research design or sampling), the com-
Research into understanding the about 50%. ment was assigned to the category best
problems that peer reviewers identify in Comments on a total of 151 manu- dictated by the context in which it ap-
research reports has barely begun. Thus, scripts were used in this study: 82 man- peared.
the goal of this study was to better un- uscripts submitted in 1997 and 72 man- Institutional review board (IRB) ap-
derstand the nature of the strengths and uscripts submitted in 1998 (three were proval was requested and granted.
weaknesses in medical education reports withdrawn, bringing the total to 69
by analyzing the ratings and comments manuscripts for that year). The con- RESULTS
made by external reviewers who rec- tents of reviews for all the manuscripts
ommended either rejection or accep- that received ‘‘questionable, probably On average, in the two periods com-
tance. A descriptive content analysis exclude’’ or ‘‘definitely exclude’’ overall bined, 4.1 reviewers evaluated each
was performed on reviewers’ comments. ratings from at least one of the review- manuscript (SD = 0.97, range 2–6).
The results should inform editors, re- ers were analyzed to identify the natures The figures were 3.9 (SD = 1.1) and 4.2
viewers, and authors of frequent and of the flaws. Conversely, the contents of (SD = 1.08) for 1997 and 1998, respec-
important reasons reviewers offered for reviews for all the manuscripts that re- tively. Reviewers were unanimous in
rejecting and accepting manuscripts. ceived unanimous approval (‘‘definitely recommending acceptance of 28 of the
These findings should also alert re- include’’ or ‘‘acceptable, probably in- 151 papers (19% overall; 23% and 13%
searchers to major flaws in conducting clude’’) were analyzed to identify the in 1997 and 1998, respectively). At
solid research. positive aspects of accepted papers. least one reviewer recommended rejec-
Lists of the reasons the reviewers tion of the remaining 123 manuscripts
METHOD gave for both positive and negative (81% overall; 77% and 87% in 1997
comments were developed as the com- and 1998, respectively). In the end, the
One data set was used for this study: ments were analyzed. A broad catego- RIME committee accepted 83 papers for
reviews of research manuscripts submit- rization scheme was used to tally the presentation in Academic Medicine (55%
ted in 1997 and 1998 for the Research reasons based on ten major categories: overall; 56% and 54% in 1997 and
in Medical Education (RIME) confer- Problem Statement (including back- 1998, respectively); 55 of the 123 man-
ence sponsored annually by the Asso- ground and literature review), Rele- uscripts that received at least one cau-
ciation of American Medical Colleges vance, Research Design, Sample and tionary exclusion initially were revised
(AAMC). The RIME manuscripts are Sampling, Instrumentation and Data by the authors and finally accepted for
peer reviewed by medical educators Collection, Results, Discussion and publication. On average, slightly over
worldwide, and those that are accepted Conclusion, Title, Abstract, and Writ- half of the reviewers (2.3 of 4.1 = 56%)
are subsequently published in a supple- ing and Presentation. Thus, the product recommended rejection of a manu-
ment to the October issue of Academic of the content analysis was two lists, script. Of the 123 manuscripts receiving
Medicine. The manuscripts are masked one containing the reasons given by the at least one recommendation for rejec-
and each is sent to four or five review- reviewers who recommended that man- tion, 15% were rejected by all the re-
ers, who write anonymous comments to uscripts be rejected (the negative com- viewers (unanimous decision), 34% by
the authors. The reviewers use a review ments) and one containing the reasons a majority of reviewers, 11% by half of

890 ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001

the reviewers, 10% by a minority, and Table 1
30% by a single reviewer. Of the re-
viewers recommending rejection, 38% Frequencies and Percentages of Reasons Given by Reviewers When Recommending
overall (29% in 1997 and 52% in 1998) Rejection of Medical Education Research Manuscripts
did not rate any of the eight review cat-
egories provided on the review form as 1997 1998 Total
‘‘unsatisfactory,’’ thus demonstrating the Category* No. (%) No. (%) No. (%)
importance of analyzing the comments. Problem statement 105 (19) 79 (16) 184 (17)
Overall, 1,053 negative comments Relevance 28 (5) 27 (5) 55 (5)
were made, and each reviewer wrote an Research design 27 (5) 35 (7) 62 (6)
average of 8.1 (SD = 5.7, range = 1– Sample and sampling 55 (10) 48 (10) 103 (10)
Instrumentation and data collection 69 (12) 76 (15) 145 (14)
30) reasons why the manuscript was
Results 105 (19) 109 (22) 214 (20)
questionable or unacceptable. During Discussion and conclusion 87 (16) 60 (12) 147 (14)
the content analysis, it was not possible Title 24 (4) 3 (1) 27 (3)
to distinguish major reasons or ‘‘fatal Abstract 6 (1) 12 (2) 18 (2)
flaws’’ from minor reasons; some of the Writing, presentation 51 (9) 47 (10) 98 (9)
negative comments were definitely less
important than were others and were TOTAL 557 496 1,053
stated mostly in an educational spirit to *A broad categorization scheme was used to tally the reviewers’ reasons into ten major categories.
help the authors. When a majority of
reviewers recommended rejection, the
number of negative comments overall
doubled (from approximately six nega- DISCUSSION while some deficiencies can be fixed
tive comments to 12) compared with within a one-to-two-month turn-
when fewer than half of the reviewers That nearly two fifths of the reviewers around time, for example, by rewriting
or a single reviewer recommended re- in this study recommended rejection of or reanalyzing some data, other defi-
jection. manuscripts but provided no unsatisfac- ciencies, such as lack of importance of
The numbers and percentages of rea- tory ratings on the review form’s check- research topics or inappropriateness of
sons given by the reviewers for rejecting list certainly reinforces editors’ requests study designs, are likely to be consid-
manuscripts are presented in Table 1 ac- for reviewers to provide written com- ered ‘‘fatal.’’ The results from the pres-
cording to broad categories and by ments in addition to numerical ratings. ent study point to six major recommen-
years. Almost three fourths of the neg- Without such comments, neither edi- dations to researchers and authors: pay
ative comments written by the review- tors nor authors can know why a man- attention to relevance (theoretical or
ers (70.1%) were categorizable to 20 uscript has been recommended for re- practical), select optimal study designs,
reasons (see Table 2). The complete list jection. select optimal instruments, select opti-
of reasons and negative comments is The overall patterns of positive and mal statistics, interpret the results hon-
presented in Appendix A. negative comments were quite similar estly, and present well-written manu-
Twenty-eight manuscripts were across the two years studied. However, scripts.
judged acceptable by all the reviewers, the diversity of the comments made by The reasons given by the reviewers in
that is, they received ‘‘definitely in- the reviewers suggested, once again, this study for rejecting manuscripts con-
clude’’ or ‘‘acceptable, probably in- that they had focused on different as- firmed Gilbert and Chubin’s list,1,p. 109
clude’’ ratings: 39% of these manu- pects of the manuscripts or weighted but are even more detailed. Also, the
scripts received ‘‘definitely include’’ their objections differently. Conse- reasons for rejecting manuscripts in this
ratings from a majority of reviewers, and quently, editors should select reviewers study were not simply mirror images of
4% by half of the reviewers; 57% of the in such a way as to strike a balance be- the reasons given for accepting manu-
manuscripts received ‘‘acceptable, prob- tween content expertise, methodologic scripts. Researchers and authors need to
ably include’’ ratings. Three fourths of expertise, and educational relevance. pay attention both to qualities of good
the positive comments written by the Some deficiencies in manuscripts can studies and good writing (e.g., relevance
reviewers (76%) were contained in nine be fixed before they are accepted for and well-crafted manuscripts) and to
reasons (see Table 3). The complete list publication, especially when RIME shortcomings of poor studies and poorly
of positive comments is presented in committee members offer their direct written manuscripts (e.g., inappropriate
Appendix B. help and guidance to authors. However, statistics and overinterpretation of the

ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 891

Table 2

Top 20 Reasons (Negative Comments) Written by the Reviewers Recommending Rejection of 123 Medical Education Manuscripts*

Reason No. % Cumulative %

Statistics: inappropriate, incomplete, or insufficiently described, etc. 118 11.2 11.2

Overinterpretation of the results 92 8.7 19.9
Inappropriate, suboptimal, insufficiently described instrument 77 7.3 27.2
Sample too small or biased 59 5.6 32.8
Text difficult to follow, to understand 41 3.9 36.7
Insufficient or incomplete problem statement 41 3.9 40.6
Inaccurate or inconsistent data reported 36 3.4 44.0
Inadequate, incomplete, inaccurate, or outdated review of the literature 33 3.1 47.1
Insufficient data presented 28 2.7 49.8
Defective tables or figures 26 2.5 52.3
Scores insufficiently reliable or unknown reliability 22 2.1 54.4
Unimportant or irrelevant topic 22 2.1 55.5
Intervention (independent variable) insufficiently described or confusing 21 2.0 58.5
Subjects insufficiently described 20 1.9 60.4
Lack of conceptual or theoretical framework 19 1.8 62.2
Underinterpretation of results; ignoring results 18 1.7 63.9
Potential confounding variables not addressed 18 1.6 65.5
Incomplete, insufficient information in abstract 17 1.6 67.1
Title not representative of the study 17 1.6 68.7
Sampling method inappropriate or insufficiently described 15 1.4 70.1

TOTAL 740/1,053

*A total of 123 of 151 manuscripts reviewed for publication in the 1997 and 1998 Research in Medical Education conference proceedings received at
least one recommendation for rejection (‘‘questionable, probably exclude’’ or ‘‘definitely exclude’’).

Table 3

Top Nine Reasons (Positive Comments) Written by the Reviewers Recommending Acceptance of Medical Education Manuscripts*

Reason No. % Cumulative %

Important, timely, relevant, critical, prevalent problem 51 20.2 20.2

Well-written manuscript (clear, straightforward, easy to follow, logical) 46 18.3 38.5
Well-designed study (appropriate, rigorous, comprehensive design) 26 10.3 48.8
Thoughtful, focused, up-to-date review of the literature 17 6.7 55.5
Sample size sufficiently large 11 4.4 59.9
Practical, useful implications 11 4.4 64.3
Interpretation took into account the limitations of the study 11 4.4 68.7
Problem well stated, formulated 9 3.6 72.3
Novel, unique approach to data analysis 9 3.6 75.9

TOTAL 191/252

*A total of 28 of 151 manuscripts reviewed for publication in the 1997 and 1998 Research in Medical Education conference proceedings received
unanimous recommendation for acceptance (‘‘definitely include’’ or ‘‘acceptable, probably include’’).

892 ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001

results). A number of strengths identi- the author’s chance of getting recom- it.’’ Scientific writing demands both
fied by the reviewers emphasized the mended for publication. conducting good science and writing
importance of researchers’ acknowledg- The detailed lists of strengths and good manuscripts.
ing the limitations of their studies (e.g., shortcomings of medical education
possible selection biases, lack of power, manuscripts reported in this study, REFERENCES
or low reliability) rather than ignoring along with other resources (such as
these deficiencies. An ‘‘honest’’ ap- Huth’s book on medical writing,4 Bor- 1. Chubin DE, Hackett, EJ. Chapter 4. Peer re-
proach to design and results in scientific dage’s paper on considerations in pre- view and the printed word. In: Chubin DE,
Hackett EJ (eds). Peerless Science: Peer Re-
writing, as noted by some reviewers, is paring a manuscript,5 and Parsell and view and U.S. Science Policy. Albany, NY:
likely to increase one’s chances of being Bligh’s guide to writing for journal pub- State University of New York Press, 1990: 83–
published. Consequently, researchers lication6), can be useful to editors and 122.
need to make a conscious effort to iden- educators in training or providing ad- 2. Council of Biology Editors. Peer Review in
tify possible biases and confounding vice to researchers and writers. The lists Scientific Publishing. Papers from the First In-
ternational Congress on Peer Review in Bio-
variables, both during the design phase can also help reviewers focus their eval- medical Publishing. Chicago, IL: Council of
of the study and once the results are in. uation of manuscripts on frequent and Biology Editors, 1991.
Researchers should ask themselves, important shortcomings. 3. Chamberlin TC. The method of multiple
‘‘What are the competing hypotheses3 In conclusion, the interdependence working hypotheses. Sci Monthly. 1944;59:
or alternative explanations and to what of science and the art of writing in pro- 357–62.
4. Huth EJ. Writing and Publishing in Medicine.
extent can they be controlled or ex- ducing good manuscripts brings to mind
3rd ed. Baltimore, MD: Williams & Wilkins,
plained?’’ two quotes from Boileau7 that are as 1999.
Many reviewers raised the issue of true today as they were over three cen- 5. Bordage G. Considerations in preparing a pub-
quality of writing (good and bad), sug- turies ago when they were written about lication paper. Teach Learn Med. 1989;1:47–
gesting that submitting well-crafted the art of poetry: ‘‘What is clearly un- 52.
6. Parsell G, Bligh J. AMEE Guide No. 17: Writ-
manuscripts is vital. Good writing is an derstood is well expressed and the words
ing for journal publication. Med Teach. 1999;
important asset in getting one’s manu- to say it come easily’’; and ‘‘Twenty 21:457–68.
script accepted, while poor writing is times on the stocks put your work, pol- 7. Boileau N. L’Art Poétique. Paris, France: Clas-
likely to annoy reviewers and decrease ishing it unceasingly and repolishing siques Larousse, 1674.

ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 893

 APPENDIX A

Reasons written by external reviewers (in Practical implications not established (13; Measurement scale insufficiently described
decreasing order within each category) 6 and 7) (11; 5 and 6)
when recommending rejection of a medical Importance not established or reported Questionable or inappropriate items on the
education manuscript submitted to the Re- (12; 8 and 4) instrument (4; 1 and 3)
search in Medical Education proceedings, Topic too narrow or simplistic (8; 4 and 4) Scoring method insufficiently described (3;
1997 and 1998. There were 1,053 negative 1 and 2)
comments, 557 in 1997 and 496 in 1998. Research design (62 total negative Example needed to understand (judge) the
comments; 27 in 1997 and 35 in nature of the variable (2; 1 and 1)
Problem statement (184 total negative 1998) Respondents not anonymous (1; 1 and
comments; 105 in 1997 and 79 in 0)
1998) Potential confounding variables not ad-
dressed (18; 8 and 10)
Insufficient, confusing, or incomplete de- Results (214 total negative comments;
No research presented (15; 5 and 10)
scription of the problem (41; 25 and 105 in 1997 and 109 in 1998)
Inappropriate or weak design (13; 10 and
16) Statistics (118; 68 and 50)
3)
Inadequate, incomplete, inaccurate, or out- Analysis insufficiently described (30; 16
Comparison group not clearly identified (5;
dated review of the literature (33; 14 and 14)
2 and 3)
and 19) Inappropriate analysis done (26; 13 and
Questionable control group (5; 1 and 4)
Intervention (independent variable) insuf- 13)
Insufficient or inappropriate timing or
ficiently described or confusing (21; Insufficient, suboptimal, or incomplete
strength of intervention (5; 1 and
10 and 11) analysis (25; 18 and 7)
4)
Lack of a conceptual or theoretical frame- Analysis not specified (21; 10 and
Questionable randomization (1; 0 and
work (19; 13 and 6) 11)
1)
Research hypothesis not stated or inappro- Incomplete analysis done (8; 7 and 1)
priate (14; 6 and 8) Too few subjects for analyses done (6; 2
Sample and sampling (103 total negative
Lack of focus, too broad (13; 8 and 5) and 4)
comments; 55 in 1997 and 48 in
Variables (independent or dependent) not P values not reported (2; 2 and 0)
1998)
identified or inappropriately labeled Inconsistencies or inaccurate data (36; 13
(9; 7 and 2) Sample size too small or biased (59; 29 and 23)
Stated purpose never pursued (7; 4 and 3) and 30) Insufficient data presented (28; 9 and
Absence of a problem statement or re- Subjects insufficiently described (20; 12 19)
search question (unable to deduce) (6; and 8) Tables and figures (26; 12 and 14)
4 and 2) Sampling method inappropriate or insuffi- Insufficient data presented (8; 5 and
Outcome (dependent) variable insuffi- ciently described (15; 9 and 6) 3)
ciently described (6; 4 and 2) Population not identified (6; 2 and 4) More needed (8; 3 and 5)
Inappropriate outcome (dependent) varia- Inappropriate sample (1; 1 and 0) Too many or redundant with text (6; 1
ble (5; 4 and 1) Sample too heterogeneous (1; 1 and 0) and 5)
Unfounded, unsubstantiated statements (3; Unequal groups (1; 1 and 0) Inappropriate format (2; 2 and 0)
0 and 3) Too complicated (2; 1 and 1)
Misleading problem statement (3; 2 and 1) Instrumentation and data collection (145 Data appear made up, unbelievable (4; 2
Unit of measurement not specified (2; 2 total negative comments; 69 in 1997 and 2)
and 0) and 76 in 1998) Data interpretation in results section (2; 1
Focusing on wrong problem (1; 1 and 0) and 1)
Inappropriate, suboptimal, or insufficiently
Outdated data (1; 1 and 0) described instrument (77; 40 and 37)
Discussion and conclusion (147 total
Insufficient or unreported reliability (22; 9
Relevance, importance (55 total negative negative comments; 87 in 1997 and
and 13)
comments; 28 in 1997 and 27 in 60 in 1998)
Untested (non-validated) instrument (13;
1998)
7 and 6) Overinterpretation of results (92; 57 and
Unimportant, irrelevant topic; adds noth- Procedure or time of administration not 35) (including conclusions not sup-
ing new (22; 10 and 12) stated (12; 4 and 8) ported by data, insufficient evidence,

894 ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001

 going beyond the data, sample, or No guidance for future studies (3; 2 and Writing, presentation (98 total negative
outcomes measured, implying causa- 1) comments; 51 in 1997 and 47 in
tion with observational studies, ignor- Ambiguity between current and past results 1998)
ing confounding variables or limita- (3; 2 and 1)
tions) Difficult to read, to follow, to understand,
Underinterpretation of results (18; 8 and Title (27 total negative comments; 24 in confusing; too much jargon (41; 19
10) 1997 and 3 in 1998) and 22)
Contradictory or conflicting assertions (10; Too long (14; 8 and 6)
Not representative of the paper (17; 14 Wrong or inaccurate terms (10; 3 and 7)
5 and 5)
and 3) Information in the wrong section, poor or-
Confusing, out-of-context interpretations
Too negative (10; 10 and 0) ganization (10; 7 and 3)
(8; 4 and 4)
Lack of theoretical framework to interpret Unedited, hasty writing, typographical er-
Abstract (18 total negative comments; 6
results (5; 4 and 1) rors (8; 4 and 4)
in 1997 and 12 in 1998)
Key points, main results don’t stand out Grammatical errors (5; 4 and 1)
(4; 3 and 1) Incomplete, insufficient information re- Inappropriate language (4; 4 and 0)
Deceptive, erroneous interpretation (4; 2 ported (17; 6 and 11) Abbreviations not spelled out (4; 1 and 3)
and 2) Inconsistent with text (1; 0 and 1) Irrelevant anecdotes (2; 1 and 1)

APPENDIX B

Reasons stated (in decreasing order within Practical, useful implications (11; 7 and 4) Innovative scoring method (2; 1 and 1)
each category) by external reviewers when Contributes to theory building, advance- Limitations of the instrument acknowledged
recommending unanimous acceptance of a ment in the field (4; 3 and 1) (1; 1 and 0)
medical education manuscript (252 total Understudied topic (2; 2 and 0) Instrument well described (1; 0 and 1)
positive comments, 165 in 1997 and 87 in
1998) Research design (27 total positive Results (19 total positive comments; 14
comments; 18 in 1997 and 9 in in 1997 and 5 in 1998)
Problem statement (36 total positive 1998)
Novel, unique approach to data analysis; in-
comments; 20 in 1997 and 16 in Well designed; appropriate, rigorous, com- tegration of multiple statistical meth-
1998) prehensive design; novel mix of designs ods (9; 6 and 3)
Thoughtful, focused, up-to-date review of (26; 17 and 9) Well thought out, appropriate analyses (3; 3
the literature; grounded, thorough (17; Well described (1; 1 and 0) and 0)
9 and 8) Easily understandable, well presented (3; 2
Problem well stated, formulated; excellent Sample and sampling (17 total positive and 1)
background (9; 6 and 3) comments; 14 in 1997 and 3 in Clear and easy-to-understand tables and fig-
Well conceived (4; 2 and 2) 1998) ures; useful, adds to comprehension (3;
Based on sound theoretical, conceptual, or Sample size sufficiently large (11; 9 and 2) 2 and 1)
educational framework (3; 1 and 2) Limitations of the sample acknowledged; se- Sufficient power (1; 1 and 0)
Clear rationale (2; 1 and 1) lection or sample bias verified (4; 4 and
Clear hypotheses (1; 1 and 0) 0) Discussion and conclusion (26 total
High response rate (2; 1 and 1) positive comments; 17 in 1997 and
Relevance, importance (68 total positive 9 in 1998)
comments; 47 in 1997 and 21 in Instrumentation and data collection (8
Interpretation took into account the limi-
1998) total positive comments; 5 in 1997
tations of the study; self criticism;
and 3 in 1998)
Important, timely, current, relevant, critical, counter-evidence, alternative explana-
appealing, prevalent problem (51; 35 Validity and (or) reliability data reported (4; tions presented; reflects scientific hon-
and 16) 3 and 1) esty (11; 9 and 2)

ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 895

Future directions discussed (4; 3 and 1) Abstract (3 total positive comments; 0 in tails, straightforward, easy to follow,
Conclusions flow from results; consistent 1997 and 3 in 1998) logical (46; 30 and 16)
with results (3; 2 and 1) Well organized (1; 1 and 0)
Confirms or extends results from previous Easy to understand (1; 0 and 1)
Good use of examples (1; 0 and 1)
studies (3; 2 and 1) Succinct (1; 0 and 1)
Practical implications discussed (2; 1 and 1) Statistical data reported (1; 0 and 1)
Argumentation well developed, compelling
(2; 0 and 2) Writing, presentation (48 total positive
Importance of nonsignificant results (1; 0 comments; 31 in 1997 and 17 in
and 1) 1998)

Title (no comment made) Well written; clear, concise yet sufficient de- 䡲

896 ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001