The Inverted-Lecture Model: A Case Study in Computer

Architecture

Edward F. Gehringer Department of Barry W. Peddycord III Department

Computer Science North Carolina of Computer Science North Carolina
State University Raleigh, NC 27606- State University Raleigh, NC 27606-
8206 [email protected] 8206 [email protected]

ABSTRACT 1. INTRODUCTION
This paper reports on an experience in using the inverted- Technology is changing the classroom. Wifi is ubiquitous.
lecture model (a.k.a. “flipping the classroom”) in computer Students come to class with a variety of electronic devices,
architecture. The first author concurrently taught two courses including laptop computers. While these devices have many
in computer architecture. One of these courses was CSC/ECE pedagogical applications, they frequently end up becoming
506: Architecture of Parallel Computers, an introductory a distraction as whole areas of the classroom can be “tuned
Graduate-level course. It was taught in a traditional format out.” [3, 12]. In response, some instructors have banned
which was mostly lecture with interspersed active-learning these devices during lecture [3], but this approach is subop-
exercises. Lecture videos were recorded to be used by timal, especially in Computer Science, as it means depriving
distance- education students; however, on-campus students students of the ability to immediately practice what they are
also had access to the recordings. The other was the being taught.
inverted CSC 456: Computer Architecture and
Active learning is an effective way to mitigate distraction,
Multiprocessing, a senior- level undergraduate course.
as students are kept engaged by answering questions, shar-
Students in this class watched the recorded CSC/ECE 506
ing their thoughts, and working problems. Active learning
lectures at home, with class time devoted to working
takes time from lecturing, but the benefits are worth it [14].
problems, writing programs, and writ- ing papers related to
The inverted-lecture model suggests that we maximize this
the course material. All “homework” was done in pairs,
benefit by using class time exclusively for active learning.
with a requirement that each student pair with each other
student at least once during the semester.
Students in the inverted-lecture class exhibited high levels 2. INVERTING LECTURES AND “HOME-
of engagement. Their performance on exams was not quite WORK”
up to the level of the students in the graduate class, but the The inverted-lecture model, also known as an “inverted
difference was not wide, and could be related to the fact that class- room” or “flipped classroom” [1], is an instructional
the students in the inverted classroom were undergraduates. approach that reverses the traditional position of lecture
From this experience, we offer observations and suggestions and home- work. In an inverted classroom, students
about inverted classes in general. watch or listen to a presentation on the day’s material
at any convenient time prior to class. During class, they
Categories and Subject Descriptors work what would usually be “homework” activities, such
as solving problems, performing experiments, or
K.3.2 [Computers in Education]: Computer and Infor-
discussing concepts with their classmates. What was once
mation Science Education—Computer Science Education
a lecture becomes more of a hands-on laboratory
environment where students learn the material both from
General Terms doing the assignments and collaborating with their peers.
Human Factors; Management In the classes of yesteryear, the only time an instructor had
available to deliver content was during the scheduled class
Keywords period. Mastery activities, therefore, had to be undertaken
at home. It is through these activities that students have
Inverted-lecture model; inverted classroom; flipped class-
a chance to practice concepts from lecture, reflect on their
room; case study
knowledge, and identify misconceptions they may have. At
home, when they come across a misconception, they have to
turn to class notes or the textbook to resolve them.
Permission to make digital or hard copies of all or part of this work for In today’s technology-rich environment, multimedia can ex-
personal or classroom use is granted without fee provided that copies are plain a concept more effectively than a talking head. Unlike
not made or distributed for profit or commercial advantage and that copies a live lecture, the presentation can be scripted and rehearsed
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific until it is just right, used across multiple semesters, and stu-
permission and/or a fee. dents can replay it until they understand. This frees up class
SIGCSE’13, March 6–9, 2013, Denver, Colorado, USA. time for collaborative learning – which works best when
Copyright © 2013 ACM 978-1-4503-1868-6/13/03...$15.00. stu-
dents work out their ideas together in front of the and misstatements can be immediately corrected. Class
instructor and any teaching assistants. In this time is convenient in this case as it ensures that the
environment, miscon- ceptions can be quickly clarified instructor and students will not have to schedule around
time commitments to meet. The other course was CSC/ECE 506: Architecture of Par-
allel Computers. It is a masters-level course also open to
3. PREVIOUS WORK advanced undergraduates. It is a required course for Elec-
The earliest systematic attempt at inverted lectures was trical and Computer Engineering graduate students and ful-
in an economics course [9]. Students were asked to read fills a “core” course requirement in “systems” (as
a topic from the textbook and either watch or listen to a opposed to “theory”) for Computer Science graduate
lecture before coming to class. Students were then given students. This course has been taught every semester for
an opportunity to ask the professor to explain points that the last several years, serving about 100 students annually.
they had trouble understanding, leading to student-focused In Spring 2012, forty-four students completed the course,
“mini-lectures,” after which students were asked to of whom 17 were distance-education students.
work problems or perform experiments. CSC 456 and CSC/ECE 506 both enroll students with sim-
While these were small courses with no more than 40 ilar backgrounds. The main difference is the presence of
stu- dents, the inverted approach has also been tried in ECE students in CSC/ECE 506; they have considerably
courses with substantially higher enrollment, e.g., a more experience in logic and hardware design. Of course,
business course that typically lectures to 300 students per the masters students are more experienced; mainly because
section was bro- ken down where students worked in most of them have worked in industry for a year or two.
groups of 24 students each [17]. All of CSC 456 students were American, but about 2/3
of the CSC/ECE 506 students were foreign nationals.
Gannod,et al. [5] identified a number of issues raised by About 85% of the course material overlapped between
flipped classes. First, they found it necessary to mandate the two classes with the following differences: The CSC
attendance, as the experiments and group work done during 456 students stud- ied pipelining and cache design in the
class time could not be replicated outside of class. They place of the CSC/ECE 506 lessons on data dependencies
required students to view the lectures before class in order and thread-level specula- tion.
to ensure they were prepared for the in-class exercises. This
can be enforced by interspersing machine-scored As with most distance-education courses in the College of
questions in the videos, or having door-quizzes when Engineering, the on-campus lectures of CSC/ECE 506 were
students arrive in class. recorded so they could be watched later by off-campus
stu- dents. Similar to work by Toto [18], students in the
One barrier to implementing inverted lectures in inverted CSC 456 class were asked to watch these
practice is the need for developing lectures and other recorded lectures. Before coming to class, students in
courseware for students. As some institutions are already both classes were re- quired to take and pass a 5-question
recording lectures for the purposes of distance education, pre-quiz over the read- ing for the day. If they got at
enterprising instruc- tors have experimented in leveraging least 4 of the 5 questions right, they received credit for
these already available resources to implement inverted the quiz. Each quiz could be retaken up to 7 times to
lectures in their own classes [18]. obtain credit. Every student was allowed to miss at least
However, many students dislike the videotaped lectures, one quiz without penalty, but for each additional quiz
some reporting that recorded lectures are not appropriate for not completed with a passing grade, they lost 0.5% from
more difficult course material [5], with others saying they their semester average. The purpose of the pre-quizzes was
are too long [18]. This has led some instructors to believe to make sure the students came to class prepared. In the
there is a need for courseware that is is specifically case of CSC/ECE 506, they needed to be prepared to do
designed with inverted teaching in mind [8]. However, active-learning exercises at several points during the class,
many reports find that students find the inverted-lecture usually 3 or 4 during a 75-minute class ses- sion. The CSC
model to be a more enjoyable learning experience, and 456 students, on the other hand, needed to be prepared to
typically express more confidence in their performance [9, work on “homework” during class.
18, 17]. Homework was similar in the two classes, including
many identical problems. The CSC 456 students
4. THE COURSES completed six problem sets, two more than the
CSC/ECE 506 students, though the 456 problem sets were
In Spring 2012, the first author taught two courses shorter so that they could be mostly finished within a
covering very similar material, the first being a senior- single class session. There were also writing assignments,
level elective, CSC 456: Computer Architecture and in which the students wrote arti- cles for a textbook
Multiprocessing. It had not been taught for several years, supplement on the class wiki. The topics included “The
due to the retirement of the previous instructor. The limits to speedup”, “On-chip interconnects,” and “TLB
subject matter is becoming more important because of the coherence,” among many others. Again, the CSC 456
need to educate students in parallelism and concurrency. students did more of these (3 vs. 2), but each individual
Moreover, parallel architecture is the most relevant aspect article was shorter than in CSC/ECE 506. Each article was
of computer architecture for CS students, because it written over three or four class periods. The articles were
affects the programming model. How- ever, only ten peer-reviewed, with the last period being used to revise their
students registered for the course, and only eight articles based on other students’ comments. The last type of
completed it. homework involved programming assignments, one on
par- allelizing a simple algorithm and the other two
simulating cache-coherence protocols. The two classes
were given iden- tical programming assignments; the CSC
456 students had two class periods to work on each.
 Table 1: Comparison between CSC 456 (inverted) and CSC/ECE 506 (traditional)
CSC 456 CSC/ECE 506
Predominant student population Senior undergraduate Beginning masters
Class size 8 44 ( 27 on-campus, 17 distance)
Class preparation Read text, pass quiz, watch video Read text, pass quiz
Class time Ask questions, do “homework” In-class active-learning exercises
Partnering requirement All assignments At least three assignments
Attendance requirement None At least 20 of the 27 classes
Actual on-campus attendance 95% 50% + virtual participants
Number of problem sets 6 4
Length of each problem set Shorter Longer
Number of programming assignments 3 3
Number of writing assignments 3 2
Length of writing assignments Shorter Longer
Number of exams 2 midterms + final 2 midterms + final

One collaborative feature of CSC 456 work was that all as- be absent. Students identified several factors contributing
signments were done in pairs, with each student to high attendance.
required to pair with all 7 other students at least once
during the semester. The rationale was that each student
1. Class time was already reserved, so it was the
comes into the class with a unique set of skills and
most convenient time for students to meet with
experience, and that by pairing with all other students,
their part- ners.
students could diffuse their skills as widely as possible.
Although CSC/ECE 506 students were only required to 2. Doing homework in class made it convenient to
work together on projects with three other students, they ask questions of the instructor and get answers
were encouraged to pair on other assignments as well. quickly.
Each CSC 456 class day began with the instructor intro-
ducing the work for the class and answering questions on 3. Students would disadvantage their partner if they skipped
it. The students then broke into pairs, with the instructor a class.
visiting each pair several times during class to see how they
4. The class was small, so students and the instructor all
were doing and answer questions or offer suggestions.
knew each other and would easily notice when
The exams in both courses consisted of two midterms and a someone was missing.
final. Two-thirds of the questions were common across the
two classes, allowing us to compare the two classes’ 5. Since the class was an elective, students only enrolled
mastery of the material. We had several hypotheses about if they were interested in the material.
how the two classes would compare.
• CSC 456 students will learn better from video Note that two of these factors (1 and 3) are dependent on
lectures, because they can replay parts. working collaboratively, one (4) is related to the class being
small, and one (5) is related to it being an elective. Factor
• CSC 456 students will be more engaged, as 2 is the only reason that would be applicable to all inverted
manifested by better attendance and more effort. classes.
• CSC 456 students will score better on exams. By and large, students came to class prepared to work on
the “homework.” Few of the questions they asked reflected
We kept track of each student’s score on each exam a lack of background knowledge. They might have prepared
ques- tion. The last class of the semester was structured as so as not to disadvantage their partner, but it seems more
a focus group. Students were asked several questions and likely that the pre-quizzes forced them to keep up with the
answered them anonymously on a Google form. The material.
answers were then displayed to the class and discussed for
a few minutes. Based on the discussion, another set of The fact that all students started on the homework at the
questions was gen- erated and posed to the students on a same time with the instructor present meant that any con-
Google form. This process was repeated three times fusion about the requirements was noticed quickly and re-
during the class period. solved before students wasted time acting on their misun-
derstandings. This helped the students make better use of
5. RESULTS class time, but perhaps more importantly, it may serve as
a model for assigning homework in an environment where
Students in the inverted class did appear to be better moti-
the use of textbook problems is increasingly risky because
vated. Attendance was approximately 95%, despite the lack
of student access to solution manuals [20, 16, 13]. If instruc-
of an attendance requirement. At least 2/3 of the time, all
tors create their own questions to use in place of “canned”
eight students were present. They skipped class only for
questions, they need a way to fix mis-specifications quickly.
important events, like interviews and illness, sometimes no-
Having a class available to do an on-the-spot requirements
tifying the instructor in advance when they were going to
inspection makes this possible.
Table 2: Exam performance between CSC 456 and
CSC/ECE 506
 CSC 456 CSC/ECE 506 Avg material
Table 3: Exposure to course How measured
in CSC 456
n = 8 students n = 44 students and CSC/ECE
CSC 456 506
Test 1, Q1 18.2 17.6
# of times book was read 18.7 (self-reported)
(3.4) (3.4) # of lectures watched 11.6 (self-reported)
∗
Test 1, Q2 6.7 12.3 Total Exposure 30.3
(4.2) (6.3)
∗∗ CSC/ECE 506
Test 1, Q6 (4) 5.10 10.4 # of times book was read 25.1 (pre-quizzes passed)
(4.4) (5.7) # of lectures watched 19.7 (in-class exercises)
Test 2, Q1 18.7 19.4 Total Exposure 44.8
(0.7) (0.8)
Test 2, Q2 18.4 18.6
(2.3) (2.4) on exams was not borne out. Of the twelve exam questions
Test 2, Q3 12.7 13.9 common to the two classes, CSC 456 students outscored
(5.9) (4.2) CSC/ECE 506 students on only two (Table 2). The aver-
∗∗ age score on common questions was 13.6 for the CSC 456
Test 2, Q5 (4) 16.1 17.0 students and 15.6 for the CSC/ECE 506 students (each ques-
(3.6) (3.9) tion was worth 20 points). Most of the difference, though,
quire a conspiracy of several students under the nose of the
instructor.
Test 2, Q6 8.3 15.5
(6.1) (5.3) Despite the engagement successes of the inverted model, our
hypothesis that students taught in this way would do better
Final exam, Q1 16.0 14.4
(4.73) (5.8)

Final exam, Q3 12.1 13.7

(3.40) (3.6)

Final exam, Q4 11.9 13.4

(4.6) (5.2)

Final exam, Q6 13.0 14.8

(2.4) (4.6)

Test 1, total 64.9 65.4

(16.4) (12.6)

Test 2, total 72.1 89.6

(13.4) (7.1)

Final exam, total 75.1 79.1

(12.5) (12.3)
∗
Students were free to skip one problem without penalty.
Only three 456 students answered question 2 on the first
midterm.
∗∗
The question numbers in parentheses are the
corresponding question numbers for the tests in CSC/ECE
506.

That said, an inverted classroom also makes it harder to

cheat in the first place. Procrastination is a major reason
that students cheat [7]; they put off starting an assignment
until it is too late, and then panic and copy from another
student or the Web. In an inverted classroom, procrasti-
nation is virtually impossible, since everyone starts at the
same time. Moreover, working collaboratively with another
person tends to diminish cheating, as has been observed in
pair-programming studies [19]. Cheating would literally re-
was on the last question of each of the three exams; if could get by by doing one or the other but not both. Not only
one excludes these questions from the computation, the were the 506 students required to pass a pre-quiz; they were
CSC 456 students averaged 15.2 and the CSC/ECE 506 also obligated to do in-class exercises during the class
students averaged 15.7. This might suggest that the CSC sessions and submit their answers on Google forms. If they
456 stu- dents knew the material nearly as well, but were failed to do this in at least 20 of the 27 class sessions, they
slower at test-taking. We must note, though, that there were pe- nalized 0.5% for each additional class missed on
was no re- quirement to attempt the questions in order. their final average. On the other hand, the CSC 456
students could probably pass the pre-quiz by either
Of course, we are comparing undergraduate students with
reading the textbook or watching the lecture.
graduate students. Any differences may reflect the greater
maturity of the grad students (most of whom had work The CSC 456 students self-reported that they read the book
ex- perience after their bachelor’s degree), or the fact that before an average of 18.7 classes in the semester, and
our graduate program is more selective than our watched an average of 11.6 lecture videos. There were 27
undergraduate program. It might also bespeak the fact classes (75 minutes each), but there was no reading
that graduate stu- dents can afford to spend more time on assigned for the first class, and the inverted-lecture model
each course, since they are typically taking 3 courses per was not used dur- ing the first week of class, so the
semester, while un- dergraduates are taking 5 or 6. students could have done the reading a maximum of 26
times and watched the lec- ture a maximum of 25 times.
An alternative explanation is that the CSC/ECE 506 stu-
The CSC/ECE 506 students passed an average of 25.1
dents had more of an opportunity to reinforce the material
quizzes based on the readings, and received attendance
in their minds, because they needed both to read the book
credit for an average of 19.7 lec- tures, based on responses
and watch the lectures, while the CSC 456 students
to in-class exercises. Computing
the totals reveals that the CSC 456 students were exposed of their group or the instructor (or TA) at any time during
to class material 30.3 times, whereas the CSC/ECE 506 stu- the class. All students thought it was helpful to some degree
dents were exposed an average of 44.8 times. By this cal- to have someone to talk to; two of them said it was very
culation, the 506 students spent 48% more time on the ma- useful. One student, though said (s)he learned more from
terial, which should have translated to higher scores. And having to “grow around challenges,” even if it took more
this may underestimate the difference, since the 456 data is time.
based on self-reporting, whereas the 506 data comes from
One innovation in this class was that every student was re-
tallying submitted work.
quired to pair with every other student in the class at least
This has two implications. First, it supports the notion that once during the semester. This was fairly easy to achieve
the CSC/ECE 506 students scored higher because they had since the class was small and there were many different as-
more time to spend on the course. If the comparison had signments. Opinions were mixed about whether this was
been between a traditional and inverted-lecture class where helpful. Three students said yes, but the others cited two
the students were taking the same number of other courses, problems: First, it was not possible to finish the homework
our findings might have been much different. Second, it in class, so they had to look for times when they could meet
raises the question of how best to combine textbook readings outside of class, and such times were not easy to find. Sec-
with lecture videos in an inverted class. Should students be ond, some students were stronger than others; it was some-
asked times a burden to work with slower students, but at least
the burden was shared roughly equally. Students were asked
• to read the textbook first, then watch the videos,
• to watch the videos first, then read the book, or
• to cover some material by reading, and other material
by watching?
Of the five students who offered an opinion one way or an-
other, four said that they would rather read the book first.
The one student who said watch the lecture first said that
(s)he could understand much of the material from the video,
and only go back to the book to study what was unclear.
Which is better for a particular student may depend on that
student’s learning style [4]. But it bears mention that at con-
ferences, when attendees have a choice of whether to watch
the presentation or read the paper first, virtually all of them
choose to watch the presentation.
In our experiment, reading the book necessarily came first
for the CSC/ECE 506 students, and that influenced the way
the material was presented in the video. Videos could in-
stead be designed to give an overview of what is written
in the textbook, and in that case, more students might be
inclined to watch first, read later. Whether the video or
the book should come first is an open question for future
research.
The inverted classroom can be a “gentler” way to learn a
subject, because students can get help from other members
how many of the other 7 students they learned from while
doing homework. The average student reported learning
from 4.9 of their 7 classmates.
Our philosophy of shuffling pairs is derived from agile
devel- opment practices, which stress the importance of
everyone learning from everyone else so that knowledge
diffuses across the group [10]. It is in contrast to team-
based learning [6], which keeps the same team together for
an extended period (often the whole semester) so that
members learn to take advantage of each other’s
strengths. We observe greater student buy-in with the
agile approach, primarily because no one feels
shortchanged for very long by having a partner who puts
forth inadequate effort.

6. OBSERVATIONS AND FUTURE WORK

Our inverted-lecture class used 75-minute videos recorded
from traditional classes for the purposes of distance educa-
tion. This is similar to work by Toto [18], as we chose to
leverage existing resources rather than creating new material
specifically for this course. Five of our eight students said
the videos should be shorter, which is reasonable, as
research shows that adult learners have a 10-15 minute
attention span at the beginning of a class [11]. While the
attention span does drop later in the class, students
watching short videos do not need to watch them in a
single sitting; they can fit them around other activities.
Shorter videos are also more likely to be watched
repeatedly to reinforce knowledge, since it is easier to find
the relevant material in shorter videos.
We asked about creating shorter videos from the recorded
lectures, but were told by the production staff that it would
require too much time. However, over the longer term, it
does make sense to record shorter videos. Students can be
asked to write the script for such videos (or even record
them) as a homework assignment. In a current class, we
are having students peer-review each other’s online lesson
videos. Beare [2] reports that students preferred videos that
were edited and touched up to mere classroom recordings.
Other researchers, e.g., Lage [9], have reported that inverted
classes take more time for students than traditional classes.
All but one of our students thought it took more time than
most other classes. Two of the eight said it took a lot more
time. Two reasons were cited: the quizzes, and the need
to meet with partners outside of class to finish assignments.
The quizzes were short (only 5 multiple-choice or checkbox
questions), so the extra time was probably mostly for re-
viewing the material to be able to pass the quiz.
At this point, we know little about how best to structure
class time. Should students be given short tasks, should
they be given tasks that they can just complete in class, or
should they be given longer assignments that can only be
started during class? In any class, students will work at
different paces, so that some finish before others. When this
happens, there are at least three choices.

1. Interrupt those who are not done; tell them the answer,
and go on to the next question.
2. Allow the faster students to work additional problems
at their own pace.
3. Structure activities so that those who got the answer
first work with others who are slower [15].
In this class, we chose the second approach, but were left In Proceedings of the 30th international conference
with the impression that the class was less interactive than on Software engineering, ICSE ’08, pages 777–786,
it might have been. New York, NY, USA, 2008. ACM.
[6] E. Gomez, D. Wu, and K. Passerini. Traditional,
hybrid and online teamwork: Lessons from the field.
7. REFERENCES Communications of the Association for Information
Systems, 25(1):33, 2009.
[1] J. Baker. The “classroom flip”. using web course
management tools to become the guide on the side. [7] D. L. R. Jones. Academic dishonesty: Are more
In 11th International Conference on College students cheating? Business Communication
Teaching and Learning, Jacksonville, FL, Quarterly, 74(2):141, 2011.
Jacksonville, FL, 2000. [8] S. Kellogg. Developing online materials to facilitate
[2] P. L. Beare. Media: The comparative effectiveness of an inverted classroom approach. In Frontiers in
videotape, audiotape, and telelecture in delivering Education Conference, 2009. FIE ’09. 39th IEEE,
continuing teacher education. American Journal of pages 1 –6, San Antonio, TX, Oct. 2009.
Distance Education, 3(2):57–66, 1989. [9] M. J. Lage, G. J. Platt, and M. Treglia. Inverting the
[3] B. Fang. From distraction to engagement: Wireless classroom: A gateway to creating an inclusive
devices in the classroom. EDUCAUSE Quarterly, learning environment. The Journal of Economic
32(4):4–9, 2009. Education, 31(1):pp. 30–43, 2000.
[4] R. Felder and J. Spurlin. Applications, reliability and [10] L. Lindstrom and R. Jeffries. Extreme programming
validity of the index of learning styles. International and agile software development methodologies.
Journal of Engineering Education, 21(1):103–112, Information Systems Management, 21(3):41–52,
2005. 2004. [11] J. Middendorf and A. Kalish. The “change-up”
[5] G. C. Gannod, J. E. Burge, and M. T. Helmick. Using in
the inverted classroom to teach software engineering. lectures. In Natl. Teach. Learn. Forum, volume 5,
pages 1–5, 1996.
[12] S. Miller and J. Connor. Student use of technology in
a large lecture. In 2010 ASEE Annual Conference
& Exposition, 2010.
[13] A. Minichiello, L. McNeill, and C. Hailey. Comparing
engineering student use of solution manuals and
Student/Faculty perceptions of academic dishonesty.
In 2012 ASEE Annual Conference & Exposition,
2012.
[14] M. Prince. Does active learning work? a review of the
research. JOURNAL OF ENGINEERING
EDUCATION-WASHINGTON-, 93:223–232, 2004.
[15] S. Robbins. Beyond clickers: using ClassQue for
multidimensional electronic classroom interaction. In
Proceedings of the 42nd ACM technical symposium
on Computer science education, pages 661–666,
2011.
[16] S. T. Santillan. Correlation between homework
solution website use and course performance.
Vancouver, BC, 2011.
[17] N. Schullery, R. Reck, and S. Schullery. Toward
solving the high enrollment, low engagement dilemma:
A case study in introductory business. International
Journal of Business, Humanities and Technology,
1(2):1–9, 2011.
[18] R. Toto and H. Nguyen. Flipping the work design in
an industrial engineering course. In Frontiers in
Education Conference, 2009. FIE ’09. 39th IEEE,
pages 1 –4, San Antonio, TX, Oct. 2009.
[19] T. Urness. Assessment using peer evaluations, random
pair assignment, and collaborative programing in CS1.
Journal of Computing Sciences in Colleges,
25(1):87–93, 2009.
[20] J. Widmann, J. Kennedy, and K. Shollenberger.
Student use of author’s textbook solution manuals:
Effect on student learning of mechanics fundamentals.
In 2007 ASEE Annual Conference & Exposition,
2007.