MIT Open Access Articles

Successful classroom deployment of

a social document annotation system

The MIT Faculty has made this article openly available. Please share
how this access benefits you. Your story matters.

Citation: Zyto, Sacha, David Karger, Mark Ackerman, and Sanjoy Mahajan. “Successful
Classroom Deployment of a Social Document Annotation System.” Proceedings of the 2012 ACM
Annual Conference on Human Factors in Computing Systems - CHI ’12 (2012), May 5–10, 2012,
Austin, Texas, USA.

As Published: http://dx.doi.org/10.1145/2207676.2208326

Persistent URL: http://hdl.handle.net/1721.1/86962

Version: Author's final manuscript: final author's manuscript post peer review, without
publisher's formatting or copy editing

Terms of use: Creative Commons Attribution-Noncommercial-Share Alike

 Successful Classroom Deployment of a Social Document
Annotation System
Sacha Zyto, David R. Karger1 Mark S. Ackerman1 Sanjoy Mahajan1
MIT CSAIL Univ of Michigan, Ann Arbor Olin College of Engineering
Cambridge, MA 02139, USA Ann Arbor, MI 48109, USA Needham, MA 02492, USA
{sacha, karger}@mit.edu [email protected] [email protected]

ABSTRACT has been unclear whether the annotation systems were too
NB is an in-place collaborative document annotation website limited, the technical ecology around them was too rudimen-
targeting students reading lecture notes and draft textbooks. tary, or the educational system was not adequately prepared.
Serving as a discussion forum in the document margins, NB Perhaps in consequence, research on the topic has lain rela-
lets users ask and answer questions about their reading mate- tively fallow for the past decade.
rial as they are reading. We describe the NB system and its
In this paper, we offer evidence that the time may be ripe
evaluation in a real class environment, where students used it
for a renewal of research and development on collaborative
to submit their reading assignments, ask questions and get or
annotation systems. We report on NB, an annotation forum
provide feedback. We show that this tool has been success-
that has been successfully deployed and used in 55 classes at
fully incorporated into numerous classes at several institu-
10 universities. Students use NB to hold threaded discussions
tions. To understand how and why, we focus on a particularly
in the margins of online class material.
successful class deployment where the instructor adapted his
teaching style to take students’ comment into account. We an- Our contribution is twofold. First, we provide evidence that
alyze the annotation practices that were observed—including the socio-technical environment of the classroom has evolved
the way geographic locality was exploited in ways unavail- to the point where the barriers that were encountered by ear-
able in traditional forums—and discuss general design impli- lier annotation tools have lowered enough to be overcome by
cations for online annotation tools in academia. motivated teachers and students. While these changed cir-
cumstances do not yet hold in all circumstances, we will ar-
Author Keywords gue that they are common enough to be worth designing for.
Hypertext; annotation; collaboration; forum; e-learning;
Our second contribution is to assess specific features of NB
ACM Classification Keywords that we believe contributed to its being adopted and valued by
H.5.2 Information Interfaces and Presentation (e.g. HCI): its users. Our design of NB’s “situated discussions,” contrast-
User Interfaces. - Graphical user interfaces. ing with the traditional “linked hypertext” model, was moti-
vated by the following design hypotheses:
General Terms • That the ability to comment in the margins, without leaving
Design; Experimentation; Human Factors; the document, would enable students to comment “in the
flow” while reading, reducing the deterrent loss of context
INTRODUCTION involved in commenting elsewhere;
Early hypertext research offered the promise of annotating
texts for educational purposes with the detailed discussion • That the in-place display of comments in the margins
necessary to understand complex material. The Web ampli- would draw students’ attention to relevant comments while
fied that promise. But it has not been fulfilled. reading, and encourage them to respond;
There is at present no collaborative annotation tool in • That the physical location of comments with their subject
widespread use in education. Past work revealed significant matter would provide a valuable organizational structure
barriers to their adoption. For example, Brush’s [3] study of distinct from the chronological organization typical of dis-
an online annotation system reported that because students cussion forums, helping students aggregate related threads
printed and read documents and comments offline, faculty and consider them together;
had to force discussion by requiring replies to comments. It
Taken together, we believed these characteristics would drive
a virtuous cycle, encouraging more students to participate
more heavily, thus providing more helpful material for other
Permission to make digital or hard copies of all or part of this work for students, yielding additional incentive to participate.
personal or classroom use is granted without fee provided that copies are 1
not made or distributed for profit or commercial advantage and that copies Zyto, Karger, and Ackerman designed and deployed NB, gathered
bear this notice and the full citation on the first page. To copy otherwise, or its usage data, analyzed it and wrote up the results. Mahajan was an
republish, to post on servers or to redistribute to lists, requires prior specific early, and to date the most successful, user of the NB system, and
permission and/or a fee. his class is the focus of our evaluation here. He was not involved in
CHI’12, May 5–10, 2012, Austin, Texas, USA. the data gathering or analysis, or authoring this article.
Copyright 2012 ACM 978-1-4503-1015-4/12/05...$10.00.
 Figure 1. NB document view. Left: Thumbnails, Center: Document (with pop-up annotation editor on the bottom), Right: Annotations

In this work, we give evidence supporting of all of our hy- discussion. Actually navigating to the discussion causes loss
potheses. We report substantial usage of NB in many classes. of context, making it harder to follow the discussion or return
To understand how and why the tool was used, we exam- to the material. A study of forum use in a class in 2002 [13]
ine one “best case” use of NB in which 91 students in a 1- found that discussion threads tended to branch and lose coher-
semester class produced over 14000 annotations. Given that ence, with many leaves of the discussion rarely read, and ob-
most of those comments had substantive content [8] and that served that “the typical nonlinear branching structure of on-
the professor and students alike praised the system, this ap- line discussion may be insufficient for the realization of truly
pears to be a successful classroom deployment of an anno- conversational modes of learning.” This was 10 years ago,
tation system. Since only limited successes have been pre- and one might believe that the current generation takes better
viously reported in HCI, hypertext, or education literature, to discussion forums. But an examination of MIT’s classroom
we assess the factors that led to this successful use and their discussion system, Stellar, showed that the 50 classes with the
implications for innovative educational uses and future text- most posts in the Spring 2010 semester produced a total of
books. 3275 posts—an average of 65.5 per class—and a maximum
of 415.2 (At the same time at MIT, one 91-student class using
MOTIVATION AND RELATED WORK NB generated over 14,000 posts.)
While there is relatively little current work, the past abounds
with studies of collaborative discussion tools for education. Improving on this “detached” situation, CaMILE [8] offered
Space limits us to projects we found most influential. It is anchor-based discussions: its HTML documents can embed
accepted that students understand material better after dis- hyperlinks from discussion anchors - places where the au-
cussing it [5, 6]. This suggests that discussion forums can be thors thought a discussion could be appropriate. Although
useful in an academic setting. Their use in this context can be this does not offer readers the flexibility to discuss arbitrary
traced back to the Plato system (1960) [4]. CSILE (1984) and points, it is a significant step towards overcoming the limi-
its successor Knowledge Forum (1995) [10] explore mecha- tations of traditional online forums by trying to situate them
nisms for encourage students to achieve knowledge building nearer the context of the document being discussed. How-
and understanding at the group level. ever, reading those annotations still requires navigating to a
different context.
These tools all support discussion of class reading materials,
but the discussions occur in a separate environment. As we 2
An important caveat is that Stellar is not a particularly good discus-
will argue below, this is a drawback: a reader might not be sion system. Recently, a forum tool called Piazzza has begun to see
aware that a topic she is considering has been discussed, so widespread adoption; we have not yet had the opportunity to analyze
might miss the opportunity to contribute to or benefit from the its usage, which clearly outperforms that of Stellar.
The WebAnn project [3] let students discuss any part of a doc- Implementation Details
ument. More significantly, it recorded annotations in-place in At the time of the study, the server-side of NB was based on
the document margins, allowing readers to see the document python, a PDF library and a postgresql database. Since then,
and the discussions on the same page. Setting the context this NB has been re-implemented using the Django framework in
way meant that comments could omit lengthy explanations order to improve portability and maintainability. NB uses a
since they would be visible at the same time as that mate- RESTful data API to exchange data between the client and
rial. The expected consequence was that a wider audience server. This allows third parties to use the NB framework and
would read and participate easily in the discussion. However, implement their own UI. The NB server is open to use by any
at the time of the WebAnn study (Spring 2001), some factors interested faculty at http://nb.mit.edu/ .
limited the benefits of the tool. Mainly, students unanimously
printed the lecture material, and worked on the printout. They
Deployment
then returned to the online site only to record the annotations
they had “planned out” on their printed copies. This intro- To date, NB has has been used by in 49 classes by 32 dis-
duced large lags between comments and replies that inhibited tinct faculty at 10 institutions including MIT, Harvard, Cali-
organic discussion, and meant that many comments arrived fornia State, U. Edinburgh, KTH Sweden, Olin College, and
too late to benefit other students while they were reading. Rochester Institute of Technology. The majority of classes
are in the physical sciences but a few are in social sciences
As people have become more comfortable online, some of the and humanities. Of the 32 faculty, 8 were using the tool for
obstacles impacting tools such as WebAnn may have shrunk. the first time this semester. Of those who started earlier, 9
With this in mind, we deployed NB to assess the present- faculty (28%) made use of the tool in multiple semesters (for
day (and future) appeal of a collaborative annotation system, a total of 18 re-uses), indicating that they have continued to
and have produced evidence that in-margin discussions can adopt it after a semester’s experience of its usage. This seems
now be an effective part of teaching. Deployed at roughly a coarse indication that they believe that the tool is helping
the same time, Van der Pol’s Annotation System [14] is an- them meet their teaching goals. Informal positive feedback
other web-based annotation framework that has been success- from many of the faculty has supported this indication.
fully used in an academic context, and was used to quantify
how both tool affordances and peer-feedback can facilitate The tool saw substantial student use in many classes. Table 1
students’ online learning conversations. shows that total number of comments submitted in the top
15 classes. 13 of these classes received more comments than
SYSTEM DESCRIPTION the maximum (415) captured in any usage of Stellar, MIT’s
NB is a web-based tool where users can read and annotate forum tool. The top five each collected more comments than
PDF documents using standard web browsers. After logging the top 50 classes using Stellar combined (3275).
in, a student typically selects a document and starts reading.
As shown on Figure 1, the document is augmented by annota- Class comments per user
tions that the students and faculty have written, which appear Approximation in Science & Eng. 14258 151
as expandable discussions on the right-hand-side panel. Hov- UI Design and Implementation (*) 10420 83
ering someplace in the document highlights the annotations Math Methods for Business (*) 4436 61
covering that place, whereas clicking somewhere on the doc- Mathematics for CS (*) 3562 23
ument scrolls to the corresponding annotations. Annotations Mathematics for CS (*) 3270 34
UI Design and Implementation (*) 2703 61
in NB are either anchored to a particular location in the doc- Signals and Systems 1996 39
ument or are general comments on the document.3 To add an Electricity and Magnetism 1254 17
annotation somewhere in the document, users click and drag Mathematics for CS (*) 1045 26
to select a region they want to comment on. This region is Pseudorandomness 880 40
Dynamics 789 21
highlighted and an in-place annotation editor pops up (bot- Adv. Quant. Research Methodology 570 9
tom of Figure 1). Math Methods for Business (*) 530 12
Concepts in Multicore Computing 336 21
Users can choose whether their comment should be visible Moral Problems and the Good Life 233 8
to everyone in the class (the default), or to the teaching staff Table 1. Usage of NB in other classes. Starred classes are re-uses by a
only, or to themselves only. The can also choose whether faculty member who had already used NB.
the comment is anonymous (the default) or signed. Once a
comment has been saved, its author can delete it or edit it
as long as there hasn’t been a reply. He can also change its USAGE ANALYSIS
properties (visibility, anonymity). Users can tag each other’s Given that NB is seeing some adoption, we wished to investi-
comments with the following tags: Favorite, Hidden, I agree, gate how and why NB is being adopted and used in the class-
I disagree, and Reply requested. room. Due to space limitations, we focus the remainder of
this article on the single most successful use of NB, in Ap-
proximations in Engineering, Spring Semester 2010 at MIT.
The teacher was Sanjoy Mahajan, our fourth author. The
3
We have found that general comments are rarely used, and do not reader might worry that we are skewing the data, but we be-
discuss them further. lieve this choice is justified for three reasons:
• Our objective here is to demonstrate, not that NB always Analysis of the open-ended comments on the user assess-
works but that NB can work in a real-world setting, which ments was done by carefully reading the comments for
shows the research direction worth pursuing. themes and patterns, as is standard practice with qualitative
data [9]. These themes were discussed by all the authors, then
• Mahajan was made an author after his usage of NB; he had re-read to examine the agreed-upon themes in more detail.
no special incentive to make the NB succeed, aside from
an interest in teaching well. FINDINGS
In this section, we assess the usage of NB by examining the
• Our data from many of the other high-usage classes is qual- corpus of annotations and its creation process. We present ev-
itatively similar, as we aim to report in an extended version idence that substantial amounts of collaborative learning [8]
of this article. occurred within NB. The annotations were primarily substan-
Approximations in Engineering had 91 undergraduate stu- tive content [8] regarding the course. Discussion threads were
dents. The thrice-weekly class lectures came from a pre- extensive. Students became active participants in questioning
print version of Mahajan’s textbook. He assigned sections and interpreting the course material, with a large majority of
of the book, usually about 5 pages long, for each lecture. questions by students answered by other students. Students
The previous four times he had taught the course, Mahajan interleaved annotation with reading, benefiting from the op-
required students to submit a paper-based “reading memo”— portunity to see content and respond to content while in the
annotations taken on the sides of the lecture pages—at the be- midst of reading, instead of navigating to a different discus-
ginning of each class. This method was popularized by Edwin sion site. Exploiting the geographic situatedness of annota-
Taylor [12]. Mahajan required students to make a “reasonable tions, students posted comments that addressed several dis-
effort”, defined in the syllabus as follows: “For reasonable ef- tinct but co-located threads simultaneously.
fort on a reading memo one comment is not enough unless it
is unusually thoughtful, while ten is too many”. Collaborative Learning
Assessing CaMILE [8], Guzdial and Turns identified 3 cri-
NB replaced the previous paper-based annotation system. teria that were deemed necessary to promote collaborative
Mahajan left the reading memo model and instructions un- learning: broad participation, sustained discussion, and fo-
changed but modified the deadline: instead of requiring that cus on class topic. We observed all three of these criteria. We
annotations be delivered in class, he made the online annota- cover each in turn.
tions due 12 hours before class, intending to peruse them prior
to lecturing (we discuss the consequences of this change in Broad Participation
the The Instructor Perspective section). There were no Teach- The 91 students created over 14000 annotations during the
ing Assistants (TAs) for this class. semester (averaging 153), while the instructor created 310.
The average number of annotations authored per student per
Method assignment was 3.67. This quantity increased over the course
Our analysis is based on log data, user questionnaires, and of the semester: a linear regression of this quantity over time
a small focus-group interview. The log data included user shows it increasing from 2.73 to 4.2 per assignment, an in-
actions down to the action level and were kept in a standard crease of 1.57 (p < 10−5 ). Although annotating was re-
log file. All annotations that users produced were stored, with quired, we take this increase over time as a sign of voluntary
the users’ consent. User questionnaires were administered at participation beyond the minimum requirement, suggesting
the end of the semester to both students and faculty. that students found the tool useful.
In total, we obtained over 1.4 million user actions and, as The instructor also posted problem sets, on which no annota-
mentioned, 14258 annotations from this class. These actions tions were required. Nonetheless, 217 annotations were made
include page seen, comment created, time spent with NB both on this material, in another demonstration of voluntary usage.
active and being ”idle”, and so on. We also obtained, to be
discussed later, questionnaires from students and interviews Sustained Discussion
with the instructor. The questionnaires consisted of Likert Of the 14258 annotations, 3426 (21.4%) were isolated
scale ratings concerning satisfaction and how NB might have threads—single annotations with no reply, while the remain-
helped or hindered understanding. In addition, they included ing 10832 (78.6%) were part of discussions—threads con-
open-ended comments about each question where they could taining an initial annotation and at least one reply. For as-
explain their ratings. signments, there were on average 13.9 discussions per page
and 3.48 annotations per discussion. As shown in Figure 2,
Analysis of the log data followed standard quantitative pro- the thread length distribution exhibits a smooth decay, with
cedures. As well, some of this data was analyzed by coding it over 400 discussion of length 5 or more, i.e. 1.4 lengthy dis-
for specific characteristics, such as being a substantive com- cussions per page of material on average.
ment, on randomly selected samples of the data. The details
of these codings and the samples are discussed in the Usage Focus on class topic
Analysis section below. The coding was done by the first We read and categorized all 413 comments in 187 discussions
author. The second and third authors reviewed the coding for a typical 5-page reading assignment (a lecture on dimen-
schemes and also the results. sional analysis, given in the middle of the term). We used
 Total questions 116
Resolved by student in same thread 59 (50.8%)
Resolved by student in different thread 14 (12%)
Resolved by faculty 10 (8.6%)
Not resolved 11 (28%)
Table 3. Breakdown of questions asked and their resolution.

asked after the assignment deadline, and 5 simply went unan-

swered. This is summarized in table 3.
In four discussions, we observed another important study
group phenomenon: Students trying to propose several hy-
Figure 2. Distribution of the number of comments per discussion potheses and look for support from their peers, often ending
their sentence with a call for confirmation (“right ?”).
Type Number Percentage Besides the 183 substantive questions and answers (50%), we
Substantive commentary 95 26% found 95 comments to the author/instructor (26%) regard-
Substantive questions 116 32.1% ing typos and suggested wording changes, and another 85
. . . about concepts 74 (23%) miscellaneous comments including brief agreements
. . . about meaning of text 42 (“me too”) and anecdotes.
Substantive answers 67 18.5%
. . . by students 57
. . . by instructor 10 Geographic Annotation
Other 85 23.4% Users of NB we able to leverage the physical placement of an-
notations in a way that could not be achieved in a traditional
Table 2. Breakdown of 363 class-learning comments (87% of the total).
forum. Of the 116 substantive questions voiced in the remain-
ing 46 discussions, we found out that 13 of them (14%) were
Guzdial and Turns’ [8] coding scheme of 6 categories in or- answered by a student, but on a nearby thread on the page.
der to label the type of comments.4 We found that annotations Each page in our sample had at least two threads that referred
related to the objectives of the course (class-learning in Guz- to another thread located nearby. In that sense, NB enabled
dial and Turns’ coding scheme) represented an overwhelming a new behavior compared to regular (i.e. non-situated) fo-
majority of the comments —363 comments (88.1%) found in rums: Participants can use the spatial proximity of threads
164 discussions (87.7%). To gain further understanding, we to implicitly address questions that were posed in the sur-
subcategorized these 363 class-learning comments. Table 2 rounding threads. In the most impressive instance, a student
summarizes their breakdown. replied to 6 surrounding questions by providing a single de-
tailed explanation of why the motion of the electron around
Question Answering the proton in the hydrogen atom can’t be described by classi-
A primary use of the tool was to ask substantive questions cal physics. Although this was explained in the textbook, the
about the material, i.e. the result of a genuine thought pro- explanation generated lots of confusion among the students
cess, stemming from an active and critical reading of the (indicated by a multitude of annotations). Those very annota-
notes: 116 comments (32.1%) in 89 discussions (55.1%). tions prompted that student to re-explain the whole reasoning
These 116 were classified as 74 (20%) requests for help to in his own terms.
understand a concept and 42 (12%) requests for clarification
about the wording in the material. Achieving such a holistic response in a traditional discus-
sion forum would be very challenging. For a student to re-
A notable result is that these occurrences included a high rate alize there were 6 distinct threads addressing the same ques-
of substantive student-to-student teaching: 57 replies (15% tion, she would have to keep a large number of discussions
of comments, 85% of total replies) in 43 (48%) discussions in working memory, or else rely on someone explicitly orga-
aimed at providing a conclusive answer were posted by stu- nizing discussions by (possibly non-obvious) topic. It’s also
dents. This was greatly appreciated by the instructor (see the unclear where the answer would go—which of the 6 relevant
The Instructor Perspective section). comments would receive the reply? And how could posters
on the other 5 threads realize that their question had been an-
Besides the student-to-student teaching, the instructor pro-
swered, again without being able to remember large chunks
vided answers in 10 discussions (11%), and 2 questions were
of the discussion forum content or relying on someone else’s
answered by their own author, leaving only 19 discussions
topical organization? The spatial layout of the notes provides
(21%) without a conclusive answer. Of these, 9 were vague
an implicit topical organization not available in traditional fo-
expressions of confusion, 2 were asked as “staff-only,” 3 were
rums, and students clearly exploited it.
4
namely: learning objectives, the technical tools (e.g programming
environment), homework (grading, strategy), the collaboration tool The geographic layout of the annotations also revealed par-
itself, infrastructure (e.g., class pace, lecture quality), and off-topic ticularly problematic parts of the text. Heavily annotated re-
(anything else). gions provided “heat maps” showing where lots of confusion
was present. Mahajan and other instructors reported exploit- In contrast, the fact that many NB users were reading online
ing this visualization to identify lecture content that needed (so getting up-to-date views of annotations) drove ongoing
clarification. discussion and rapid responses. Students using NB particu-
larly appreciated the fact that they could read, comment, and
reply all at the same time, and get clarification on confusing
Tagging
points in the lecture notes in a timely fashion (cf. the “Student
As was observed in the context of the usage of digital ink [1], feedback” section). NB yielded a much greater proportion of
comments were often used to tag a section in the text with la- replies than WebAnn, without imposing WebAnn’s differen-
bels such as “confusing”, “like”, “dislike”, and “easy”. Those tial deadlines or specific requirement to reply.
comments used lots of screen real estate to convey small bits
of information, sometimes obscuring more substantive com-
ments. Still, students reported that it was very useful to tag
and see others’ tags.
Examining comments of 5 words or less, we found that
375 of them (2.7% of the total) could be replaced by one
of the following 8 tags without loss of meaning: I agree,
typo, cool/interesting/amazing, confusing, thanks, lol/funny,
me too, what’s this ?. A tagging interface could have pre-
sented this information in less cluttered and more informative
form, e.g. by color coding. Figure 3. Distribution of intervals (in hours) between the comments cre-
ation time and the corresponding assignment deadline
Continuous Ongoing Discussion
Although the number of assignments in our class differed The ongoing nature of the interaction is confirmed by Figure
from the WebAnn experiment [3], we found that the number 3, which presents the number of comments posted as a func-
of annotations per author per assignment were very similar: a tion of the time (in hours) between a comment creation time
bit more than 4.5 However, these annotations classify differ- and the deadline for the corresponding assignment (10PM on
ently than in WebAnn: the larger number of replies per author the day before lecture). We can observe 3 main clusters,
per assignment (2.53 vs 1.58 in WebAnn) indicates that stu- corresponding to annotations authored by students who be-
dents who used NB engaged in more conversations with one gan working on their assignments respectively 2 days before
another. This difference is even more notable given that the (1047 annotations i.e. 7.7%), 1 day before (2682, i.e. 19.7%)
WebAnn experiment required each student to enter at least and on the due date (7344, i.e. 53.7%). The remaining com-
one reply per assignment, whereas the class using NB had no ments (2599, i.e. 19%) were authored mostly later, either as
such requirement. part of extensions, or when a old discussion was revived, typ-
ically before an exam.
One possible explanation for this difference might be the dif-
ference in online versus offline usage of the two tools. NB In summary, Figure 3 shows that NB participants didn’t ex-
users rarely printed the lecture notes—our end of class poll perience the problem of discussion seeding that WebAnn did
estimated only 16.9% (N=26 and SE=5.16) ever did so. In - i.e. assignments done right before the deadline, which pro-
contrast, WebAnn users printed lecture notes systematically. duce rushed single comments rather than helpful discussions.
Common practice (cf. [3], p. 4) was to print and annotate a Clearly, there is a peak of activity in the few hours before
paper copy of the notes, and at some later convenient time the deadline, but since many comments have been entered al-
“transfer” the annotations online. There are plausible ratio- ready, there are many opportunities for discussion. In fact,
nalizations for this offline usage. WebAnn users lacked ubiq- even annotations entered by “early-bird” students 2 days be-
uitous access to the Internet and the WebAnn software (which fore the deadline were spread out enough to enable discus-
involved a special browser plug-in). The user experience with sions on that very same day: 39% of comments entered on
2001-vintage Web applications was poor, and students had that day were replies.
less experience working online.
Regardless of the reason, WebAnn’s offline usage created a Annotating in the Flow
large lag between the time an annotation was first recorded A strong motivation for our design of NB was the hypoth-
(on paper) and when it could be read and a reply generated. esis that discussion can be improved if it is situated in the
And students who printed too early might never see some context of the document. Letting readers comment without
comments at all. To address the problem, Brush et al. [3] leaving the reading environment meets the goals of keeping
found it necessary to enforce two separate deadlines: Tues- the user “in the flow” of their work, rather than interrupting
days at noon for submitting initial comments, and Wednes- it [2]. It also means that readers can encounter and respond to
days before class for (required) replies. comments and question as they read, instead of having to go
hunting for relevant comments.
5
In a study of how peer-feedback can increase the relevance of on-
line discussions, van der Pol [14](chapter 4) also reported 4.7 anno- Given this hypothesis, we tried to measure whether such “in-
tations per student per (weekly) assignment. flow” annotation happened. More specifically, we looked
whether opportunistic replying occurred, namely writing a re- This resulted in analyzing a set of 3826 replies, for which we
ply to a comment while reading the lecture notes. We took found a linear regression slope of 0.47 (p < 10−15 ), and a
two approaches. adjusted R2 = 0.1125. This implies that a statistically very
significant portion of the user’s placement of replies can be
Our first approach considered the distribution of annotation “explained” by the user placing them at the position indicated
times over a “reading session”, i.e. over a single time span by a linear read through the text.
that users would spend when doing their reading online in or-
der to prepare for lecture. We used log data to identify the
USER ASSESSMENT
beginnings and ends of sessions. We focused attention on
sessions of length between ten minutes and one hour, assum- The general utility of NB was also demonstrated in student
ing that shorter sessions may have reflected quick look-ups and faculty feedback. Students reported that using NB helped
of specific bits of information, and longer sessions may have them learn. They felt the level of class discussion to be quite
included substantial multitasking or idle time or logging er- high and valuable to them in understanding. Anchoring the
rors. We looked at the 6544 annotations that were made dur- discussion in the material motivated students to return to the
ing those typical reading sessions. We scaled the times of material, which they argued benefited their learning. The in-
those annotations as a fraction of the total time spent reading structor reported that NB helped him to teach better and also
and plotted the distribution. Overall, this distribution is flat, observed that it let students be involved in a genuine discus-
showing that annotations were being authored throughout the sion while trying to understand the material.
course of typical reading sessions. We did the same for the
subset consisting of 3676 replies, and found that it too was Student feedback
flat, suggesting that readers were replying to comments in the At the end of the term, students were asked to fill in an op-
midst of reading. Figure 4 shows this distribution for replies tional web-based poll. We wanted to know more about their
(the distribution for annotations is similar). annotating practices (for example, whether they print the ma-
terial or annotate while reading it online) and how NB had
helped or hindered their understanding of the material. Of
91 students, 37 (40%) responded. However, not all students
completed the survey, so we report varying N ’s below.
Overall, students valued NB. They were asked how they felt
that NB had impacted their learning during the term, on a 5-
point scale (1: very positively to 5: very negatively). The
response was positive with a mean of 1.72 (N=37).
We also analyzed the comments that accompanied the ratings.
We found three themes:
Significant Discussion and Learning
First, students appreciated seeing others’ efforts, including
the answers to their own questions by other students but also
questions asked by peers. Some students felt that they were
engaged in a helpful discussion about the material:
• Never had this level of in-depth discussion before. . .
• It was cool to see what [sic] other people’s comments on
Figure 4. Distribution of relative creation times for replies over the
course of a reading session.
the material.
• I really enjoyed the collaborative learning. The comments
Our second approach considered reading activity on single that were made really helped my understanding of some of
pages, and determined whether the (relative) time a reply was the material.
authored was linearly related to the position of the thread on
Students liked being able to get questions answered in timely
that page, which would suggest that replies were written as
fashion:
the reader traversed from beginning to end of the page.
• I was able to share ideas and have my questions answered
Again, we normalized the time of writing as a fraction of the
by classmates
total time spent reading each page (we logged entries and ex-
its to each page), and correlated that normalized time to the • Open questions to a whole class are incredibly useful. Ev-
position of the annotation on the page (all readings in the class eryone has their area of expertise and this is access to ev-
were single-column, so reading ran linearly from top to bot- eryone’s combined intelligence
tom). We filtered out pages where students spent less than 10
seconds or more than an hour, and data points where the nor- • Due to the considerable number of people in the class and
malized time wasn’t in the [0, 1] range (due to measurement the requirement to make annotations, responses are prompt
errors such as clock differences between client and server). and predominantly helpful
This led to a general sense that NB allowed much more inter- The Instructor Perspective
activity in the reading: We interviewed the course instructor, Sanjoy Mahajan, to un-
derstand his motivations and practices while using NB. Ma-
• The volume of discussion and feedback was much greater
hajan reported that the impact of NB on his class was very
than in any other class.
positive. Conversely, we speculate that some of the success
The student-to-student teaching as well as automatic email that NB had in his class is due to the way Mahajan modified
notifications when an reply was posted seemed to make his teaching practices to take advantage of NB.
the feedback time acceptable: On a scale ranging from 1
(strongly agree) to 7 (strongly disagree), students reported an Adapting to NB
average of 3.04 (N=27), i.e. “Somewhat agree” to the state- Guzdial and Turns [8] urged exploring how the instructor’s
ment “When I ask a question using NB, I usually get a timely involvement impacts “. . . his or her willingness to explore fur-
reply”. ther uses of information technology and to participate in ed-
ucational reform”. One possible reason that NB worked so
Situated Annotations well in this class could be that Mahajan adjusted his teach-
Although the comments above show that students appreciated ing style to exploit NB. As we discussed in the opening of
the in-depth discussions, these could equally have taken place the Usage Analysis section, Mahajan had already incorpo-
in a traditional forum (though they often do not). However, rated a “reading memo” practice into his class. He thus had
other comments showed how students specifically valued the a sense of how to motivate students to make annotations as
situating of the discussion on the text: well as how to take advantage of them.
• The commenting system on NB is really useful because it Mahajan required use of NB, but his requirement were de-
allows us to challenge the text and each other and to see liberately vague: students had to submit one or more com-
feedback from others taking the class. ments that showed “decent effort”. This was guaranteed to
receive full credit, regardless of whether the author was right
• Being able to read the comments of others allows me to or wrong. Students had to provide a steady effort by com-
review the text more than once based on these comments menting on every lecture, but were automatically allowed up
The first quote, referring to “challenging the text,” shows how to eight extensions of 1 week each. Two students interviewed
the primary material was kept central to the discussion, unlike in our focus group indicated that since they didn’t know what
in a separate discussion forum. The second emphasizes the “decent effort” really meant, they used their common sense
role of comments that are present while reviewing the text. in order to participate in a “decent way” (i.e. contribute an
interesting participation given their other time constraints).
Indeed, students felt that NB provided additional motivation
to do the readings and interact with them: Mahajan also emphasized to the students that unlike prob-
lem sets, where faculty are assessing whether students get the
• [NB] forced me to read the text and interact with it. right answer, student annotations were assessments of how
• It forced me to read the ”textbook” which I don’t usually well he was doing as an explainer. This created an atmo-
do. It forced the professor to break it down into chunks, sphere where students valued the chance to make comments
making material more concise and less repetitive/tedious on material written by the faculty.

Understanding where problems are Feedback

Earlier we discussed the “heat map” effect of seeing where The WebAnn study [3] reported that on-line comments often
comments cluster densely. Students were asked to rate competed with in-class discussions. Mahajan observed the
whether NB helped them understand where their classmates opposite: he explained that NB was an unprecedented suc-
had a problem on a 7-point Likert scale (1:strongly agree, 7: cess for his class, because he was now able to adjust the con-
strongly disagree). The class had agreement that this was of- tents of his upcoming lecture in order to address the confusing
ten true (mean=2.03, N = 28). points mentioned on NB. Comments were due at 10pm on the
Open-ended answers to this question also provided evidence day before the lecture. He would begin reading them around
that students found their ability to see the confusion of others 11pm and adapt the material in time for his lecture starting at
to be helpful for self-assessment: 11am the following day. He reported that the sheer amount
of page-flipping would have made this impossible using his
• It’s encouraging to see if I’m not the only one confused previous paper-based submission approach. In the sample
and nice when people answer my questions. I also like lecture we analyzed, we found 3 requests to use simpler ex-
answering other people’s questions. amples, 2 requests to review/explain a concept during class
(Mahajan replied that he would try), and 4 notes mentioning
• . . . [NB] helps me see whether the questions I have are rea- something that had been seen in class. In-forum and in-class
sonable/shared by others, or in some cases, whether I have contents seemed to complement each other.
misunderstood or glossed over an important concept.
Finally, Mahajan mentioned that the “part that [he had] un-
derestimated about NB”, and which “turned out to be really
important” was the extent at which students answered each
other, which is why he only needed to participate in 10.4%
of discussions. This connects with our discussion above, that requirements and guidelines that aren’t too strict (i.e. reason-
students found responses timely. able and steady effort vs. a required number of comments
per week). In another class, where the requirement was set
DISCUSSION AND FUTURE WORK as “exactly two annotations per lecture,” the students met that
NB has provided evidence that an in-place social annotation requirement exactly and never exceeded it. On the other hand,
tool can be adopted and considered of positive educational another class where annotations were not required at all did
value by both faculty and students in a modern classroom. nonetheless see substantial usage. Clearly the question of ef-
In an attempt to understand how and why this adoption takes fective motivation to annotate requires further investigation.
place, we have centered our analysis on showing that NB pro-
moted student-to-student teaching; and that NB’s in-place na- A Finished Product versus a Process
ture encouraged integrating annotations during reading, mak- As some classes have begun to use NB several times, an inter-
ing WebAnn’s enforcement of separate deadlines for com- esting question has emerged about whether or not keep previ-
ments and replies no longer necessary. Here we discuss ram- ous terms’ annotations available for future student use. To the
ifications and interesting open issues. extent that these annotations clarify the material, it seems nat-
ural to preserve the “improved” lectures plus annotations for
Is Data from a Single Class Convincing? the next group’s use. In practice, faculty users of NB invari-
Clearly, we benefited from an very talented and motivated ably discard the old annotations. They say that the process of
faculty user of our system. One might fairly ask whether discussing the notes in the margins is considered a valuable
“other” faculty could expect to see any of the same benefits. contribution to the students’ learning, which would be lost if
While a detailed analysis of how different faculty affect out- past comments were already available for reading.
comes must await a future paper, Table 1 demonstrates that
many other faculty at several other institutions were able to At the same time, marginal notes can provide an effective
achieve significant adoption, some approaching the best case contribution to a text’s narrative. Knuth’s Concrete Mathe-
studied in this paper, even though few of them had previously matics [7], a traditional textbook, publishes in its margins a
made use of reading memo requirements. We cannot yet re- selection of the marginal comments recorded by students us-
port whether adoption in these other classes was determined ing a draft in the first version of the class. These comments
by the same factors as the one analyzed here, or entirely dif- add insight, humor, and unique student perspectives without
ferent ones. At a high level, however, we can confirm that disturbing the main narrative. We believe there would be
numerous faculty believed that the tool was a useful enhance- value in tools that help instructors to curate annotations, se-
ment to their teaching practice. lecting some to drive changes in the text, some that would
be most valuable remaining as marginal notes, and some that
Of course, some preconditions apply to successful usage of should be removed so that future classes can rediscover them.
NB. As one reviewer noted, “Their technology is good for
students in highly connected environments who all have com-
puters and for teachers who are tech savvy and lecture using Better Support for Annotation’s Specific Affordances
online materials rather than a textbook. As a counter exam- Our users discovered and exploited certain capabilities of an-
ple, the tweedy old-school professors at my husband’s less notation that are not present in traditional forums. We can
than super-tech-savvy graduate school who all use textbooks provide better support for those capabilities. Above, we dis-
would not be a good target for this technology.” However, cussed how geographic annotation was leveraged to answer
we believe that the necessary preconditions are already quite sometimes-multiple questions in other threads. It would be
common and becoming more so. useful to capture this answering behavior in the thread struc-
ture, for example to let an author explicitly mark (multiple)
Adoption versus Learning Outcomes threads to which they were responding. We also discussed the
The Holy Grail of an educational tool is improved learning use of annotations as tags, and suggested there could be value
outcomes. Assessing learning outcomes is always difficult. in directly supporting tags presentation through less cluttered
Here, we settled for assessing adoption by faculty and, secon- and more informative interfaces such as color coding.
darily, students. Numerous faculty have voluntarily adopted
the tool, and numerous students have gone beyond the re- CONCLUSION
quirements in using it. It is conceivable that all these faculty Our development of NB was driven by several design hy-
and students are misguided, and that NB is not in fact en- potheses about the way an “in-place” annotation tool could
hancing learning outcomes. However, we feel that so many outperform traditional forums as a medium for discussion of
faculty and students are likely on to something, suggesting classroom materials. Situating discussions in-place allows
that improved learning is happening. students to annotate and question while reading, remaining
in the flow instead of losing context on a different forum. It
The Importance of Class Requirements draws student attention to relevant discussion at the moment
Effective use of a social annotation system in the classroom they are reading the material, instead of requiring them to
isn’t only about designing the right tool; motivating usage is consider that there might be relevant discussion and search
also essential. Mahajan believes that annotations need to be for it (and retain the context) in a separate environment. It
a requirement. He believes that a key aspect is to have usage allows them to consider all relevant discussion threads drawn
together by physical proximity, instead of organized by post- in the expectations of users of that technology cannot be
ing chronology, and author answers that draw many of these clearly worked out from this single case study. However, the
threads together. evidence suggests that we have reached a turning point where
online social annotation systems could become a standard and
Our deployment of NB has provided evidence supporting valuable educational tool.
these hypotheses. In our “best-use” class, students con-
tributed 14,000 distinct annotations, outdoing by a factor of ACKNOWLEDGEMENTS
4 the combined product of the 50 most active classroom dis-
This work was supported by the d’Arbeloff fund for Excel-
cussion forums at the same university. Students and faculty
lence in Education. We thank Michael Bernstein, Katrina
gave significant positive feedback regarding the role of NB
Panovich, Rob Miller, mc schraefel, and the Haystack group
in the class. Data show that students write and read com-
at MIT CSAIL for their help in this work and our anonymous
ments in tandem with reading the primary materials, and ex-
reviewers for their useful feedback.
ploit the geographical coherence of annotation to draw multi-
ple threads together into substantive discussions.
REFERENCES
From our experience we were able to draw the following de- 1. Anderson, R. J., Hoyer, C., Wolfman, S. A., and
sign conclusions: Anderson, R. A study of digital ink in lecture
presentation. In Proceedings of CHI ’04, ACM (New
• Current students do abandon paper for online reading. We York, NY, USA, 2004), 567–574.
hypothesize that the gain of interactivity (access to the lat-
est comments, asking a question while reading) outweighs 2. Bederson, B. B. Interfaces for staying in the flow.
the irreplaceable affordances of paper as a support for read- Ubiquity 2004 (September 2004), 1–1.
ing, described in [11]. 3. Bernheim Brush, A. J., Bargeron, D., Grudin, J.,
• Students interleave annotation with reading, implying that Borning, A., and Gupta, A. Supporting interaction
it must be kept easy to annotate while in the flow of read- outside of class: Anchored discussion vs. bulletin
ing. For instance, we recommend against using modes or boards. In Proceedings of CSCL 2002 (2002), 425–434.
required fields. 4. Blitzer, D. The wide world of computer-based
education. Advances in computers 15 (1976), 239–283.
• Students combine response to several geographically co-
located threads, implying that future tools should support 5. Bonwell, C. C., and Eison, J. A. Active learning,
marking multiple threads for simultaneous reply. creating excitement in the classroom. ERIC
Clearinghouse on Higher Education (1991).
• Requiring annotations may be necessary at least at the be-
ginning of the term, but students learn to value them and 6. Chickering, A. W., and Gamson, Z. F. Seven principles
go far beyond the requirement. for good practice in undergraduate education. American
Association for Higher Education Bulletin (1987).
• There is demand among faculty for a tool to stimulate stu-
7. Graham, R., Knuth, D., and Patashnik, O. Concrete
dent feedback and discussion. Feedback can happen at a
Mathematics: A Foundation for Computer Science,
timescale that allows adapting the following lecture based
2nd ed. Addison-Wesley, 1994.
on the questions and comments from the previous lecture
and the reading assignment. 8. Guzdial, M., and Turns, J. Effective discussion through a
computer-mediated anchored forum. The Journal of the
• Student-to-student feedback is far faster than faculty feed- Learning Sciences 9, 4 (2000), 437–469.
back. Students overwhelmingly appreciate that fast re-
sponse time. The design implication is that students should 9. Miles, M. B., and Huberman, A. M. Qualitative Data
be able to discover questions that are currently being asked. Analysis, 2nd ed. Sage, Thousand Oaks, CA, 1994.
Future system should help students differentiate between 10. Scardamalia, M. Csile / knowledge forum. In Education
”stale” conversations and the ones that are worth reading. and Technology: An encyclopedia (2004), 183–192.
• Previous studies argued that adoption barriers prevent on- 11. Sellen, A., and Harper, R. The Myth Of The Paperless
line annotation. We refute this. Yet, instructors should be Office. MIT Press, 2001.
made aware that such online communal annotation tools
aren’t a one-size-fits-all solution. This paper’s best case is 12. Taylor, E. F. Guest comment: Only the student knows.
an example where it worked wonderfully, but future work American Journal of Physics 60, 3 (1992), 201–202.
will need to uncover when and why it does and does not 13. Thomas, M. Learning within incoherent structures: The
through comparative longitudinal studies. space of online discussion forums. Journal of Computer
NB offers an “existence proof” that it is possible for an on- Assisted Learning 18, 3 (Sept. 2002), 351–366.
line collaborative lecture-note annotation system to succeed 14. van der Pol, J. Facilitating Online Learning
in a classroom setting. This contrasts with experience using Conversations. PhD thesis, Universiteit Utrecht, 2007.
the technology of previous decades. Whether this is due to
changes in teaching style, changes in technology, or changes