See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/338659877

Typing Exercises as Interactive Worked Examples for Deliberate Practice in CS

Courses

Conference Paper · February 2020

DOI: 10.1145/3373165.3373177

CITATIONS READS
6 487

5 authors, including:

Adam Matthew Gaweda

North Carolina State University
10 PUBLICATIONS   12 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Adam Matthew Gaweda on 11 February 2020.

The user has requested enhancement of the downloaded file.

 Typing Exercises as Interactive
Worked Examples for Deliberate Practice in CS Courses
Adam M. Gaweda, Collin F. Lynch, Nathan Seamon, Gabriel Silva de Oliveira, Alay Deliwa
agaweda,cflynch,nseamon,gsilvad,[email protected]
North Carolina State University
Raleigh, North Carolina, U.S.A.
ABSTRACT 1 INTRODUCTION
Solving complex Computer Science problems require students to uti- Most people are comfortable with the flexible grammatical structure
lize both higher level skills like problem translation, decomposition, of their native language. Similarly, while second language learning
and implementation, and lower level skills such as implementing requires mastering novel syntax and grammar (e.g. ñ л 未) these
solution patterns and debugging. However, novices often struggle languages are flexible and often map to the learner’s first language. In
with with these lower level skills, preventing them from moving contrast to informal languages, programming languages have rigid
forward. As a consequence, they risk developing a shallow under- syntactic structures which require absolute precision for correct
standing of code concepts and fail to practice crucial higher-level usage (e.g. int i=0; i=1; i++; i==1). Any error, however small,
skills. We assert that if students were given the ability to practice will produce different output or no output at all. Computer Science
these lower level skills outside of traditional programming exercises, originates from logic and mathematics, and thus are taught in a sim-
they would be more apt to recognize and resolve them. For this we ilar fashion. However, a recent fMRI study showed both formal and
present our analysis of TYPOS, a platform for low-level skill practice informal language learning activate many of the same regions in the
using interactive worked examples as typing exercises. In this work brain and in fact program comprehension is more closely associated
we analyzed self-selected students based on their platform usage with informal language processes than to logic and mathematical
and the differences in course performance and behavior between analysis [39]. Regardless, computer programming is a necessary
groups. We found that students in a college level CS course that reg- requirement for Computer Science students and so if a student is
ularly completed typing exercises earned higher course grades and unable to “speak the language”, they will struggle in their CS courses.
higher learning gains than those who worked irregularly or not at This high precision is often the first challenge that novices en-
all. Students who actively used the system and completed typing ex- counter when learning a programming language [2, 7, 40, 42]. Fur-
ercises made fewer compilation errors on graded projects. Students ther, students that struggle initially in a CS course will continue
did not, however, use the tool while working on their projects but to struggle as the course progresses [1]. This experience of facing
engaged in deliberate practice sessions prior to lecture. We argue this immediate errors is extremely discouraging for students and can act
model provides students with the benefits (although with marginal as a barrier for students entering CS and thus reduce access to the
significance) that accrue from worked examples as well as deliberate discipline [29, 43]. Therefore, reducing such barriers and increasing
practice on a key area of struggle for novice students. persistence and retention is essential.
When faced with difficulties students often fall back on refer-
CCS CONCEPTS encing or simply pasting in code from course examples or third
• Theory of computation → Active learning; • Applied com- party websites, the latter of which may invoke academic integrity
puting → Interactive learning environments. violations. Referencing examples is not necessarily malicious. Com-
paring example solutions has been shown to help students form
KEYWORDS more abstract mental models in STEM domains such as mathematics,
physics, and block-based computer programming [12, 41, 57]. More-
typing exercises, novel exercises, deliberate practice, spaced practice, over, prior research on CS problem solving has shown that solvers
activity sequences use “recurring basic plans” [48] to solve problems, and professional
ACM Reference Format: developers routinely reuse code in their work.
Adam M. Gaweda, Collin F. Lynch, Nathan Seamon, Gabriel Silva de Oliveira, However, there is a fundamental difference between an experi-
Alay Deliwa. 2020. Typing Exercises as Interactive Worked Examples for enced professional planning to use their own well-tested code, and
Deliberate Practice in CS Courses. In Proceedings of the Twenty-Second Aus- a novice pasting a poorly-understood example. Students who copy
tralasian Computing Education Conference (ACE’20), February 3–7, 2020, Mel- and paste in lieu of problem-solving risk creating shallow mental
bourne, VIC, Australia. ACM, New York, NY, USA, 9 pages. https://doi.org/ models of the problem at hand because they do not take the time to
10.1145/3373165.3373177
fully understand the essential CS concepts or the code that they are
using. They also risk introducing errors or following unproductive
Permission to make digital or hard copies of all or part of this work for personal or paths which will impose additional unnecessary work, a process
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation known as floundering, and missing opportunities to practice essential
on the first page. Copyrights for components of this work owned by others than the higher-level skills.
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
Our overall research goals are to provide instructional support
and/or a fee. Request permissions from [email protected]. for students in CS through the use of simple exercises involving in-
ACE’20, February 3–7, 2020, Melbourne, VIC, Australia teractive worked examples. These exercises are designed to provide
© 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. students with opportunities for deliberate practice [18] of problem
ACM ISBN 978-1-4503-7686-0/20/02. . . $15.00
https://doi.org/10.1145/3373165.3373177 solving skills while mitigating the risk that they will copy poorly

105
ACE’20, February 3–7, 2020, Melbourne, VIC, Australia Adam M. Gaweda, Collin F. Lynch, Nathan Seamon, Gabriel Silva de Oliveira, Alay Deliwa

understood code and allowing them to focus on low-level skills in of subgoals or subtasks which are in turn decomposed into base cases,
realistic code. Our hope is that students will process these examples which are implemented directly [28]. However, novices face difficulty
in a more active manner [13] than passively reviewing them. In effect identifying the appropriate subgoals for their specific problems [55].
these exercises provide students with a productive alternative to ran- For example, to implement the Monte Carlo algorithm [34] for
dom copying and pasting by helping them to accumulate their own approximating the probability of rolling two or more dice (shown
repository of recurring basic plans while reinforcing their ability to in Figure 1), students need to implement the following subgoals:
implement said plans.
• create a running total of successful trials
In this study we analyzed the effect isolating syntax error resolu-
• create a loop for trials
tion through typing exercises would have on student performance.
We chose typing exercises for this analysis because they mirror ac- – create an inner loop based on how many dice will
tive learning activities presented in the ICAP framework [13], syntax be rolled
errors as often the first barrier novice CS students experience [2],
∗ generate that many random numbers
and based on studies finding that lower performing students strug-
∗ if the numbers all equal the expected number,
gle with higher level meta-cognitive skills, such as self-explanation
increment the running total
[24, 57]. We assert this indicates that novices need to practice these
lower levels skills in order to build a foundation of understanding
• divide the running total with the number of trials
CS concepts and once established will allow them to move toward
those necessary higher level skills. Once a novice has identified a list of subgoals, they still face chal-
Rather than present a complete example for review or allowing the lenges implementing those goals into code [2, 7, 8, 40, 42, 55, 55].
students to click through it by line by line or step by step as is common Individual mistakes in syntax, semantic structure, or the runtime
in other domains (e.g. [12, 33]), we require students to re-enter the logic will cause the code to fail and trigger explicit error messages or
example themselves. Our approach provides dual benefits: first, the silent bugs. Thus, students need to alternate between implementing
examples illustrate the application of general concepts (e.g. Iteration, new goals and identifying and rectifying errors before they can move
Monte Carlo, Design Patterns, Finite State Machines, etc.) to help forward with other subgoal completion [2].
build their “recurring basic plans” repository; second, they provide However, repeatedly encountering syntax errors is discouraging,
students with the opportunity to practice low-level implementation despite having a firm understanding of subgoals. When faced with
skills. Moreover, this approach requires them to review code on a repeated errors, students may halt any further progress due to two
finer level than they would through simple copying and pasting [13]. states: “wheel spinning”, losing sight of their overall plan as they
To analyze this approach, we developed a web-based platform resolve minor errors [6], or “helplessness”, where the student quits
called TYPOS which we have incorporated into a second semester attempting the problem altogether [17, 22, 37]. In the context of de-
introductory CS course at a four year university in the southeastern bugging, this would mean that students either quit fixing their bugs,
United States for three semesters. TYPOS is an exercise platform de- get lost resolving bugs and lose sight of their solution plan, or simply
signed to provide students with learning activities to help teach them start over. Additionally, programming exercises are non-trivial prob-
CS topics. During the study, 331 self-selected students volunteered to lems that may require multiple hours to complete, and it is difficult
use the system, 77 of whom did so regularly. We examined how these to identify milestones for progress in a programming exercise. Even
active students used TYPOS in the class and how that use correlated after resolving an error, students may be no closer to completing the
with their other work. We classified the students into three groups solution, despite the extra time spent doing so. Students may view
based upon their work habits, collected the code that students sub- this as “wasted” time and only further their beliefs that Computer
mitted for their regular coding assignments, and analyzed their use Science is “not for them” [17].
of the online development platform, GitHub, to understand how they Resolving bugs is essential to learning in CS and imposes ad-
integrated work on TYPOS with their other tasks. We also collected ditional difficulties on students. Youngs claimed that debugging
their course and exam grades to assess how their work on TYPOS semantic errors occupies the majority of beginners’ time [56]. How-
related to their overall performance and conceptual understanding. ever, Brooks argued the opposite; speculating their reported low
This work builds upon our prior preliminary studies [23] which debugging time was due to smaller sized programs not requiring
focused on their behaviors on the TYPOS platform. as much debugging time or over the availability of “canned” code
Our goal in this paper is to answer the following research ques- [10]. Both reports used languages that are no longer common in
tions (RQs): introductory CS courses (e.g. FORTRAN & COBOL), however both
RQ1 How do different patterns of use correlate with indicate debugging a part of code development. The speculation of
student performance? “canned’ code supports our focus on providing additional examples
RQ2 Do students refer to TYPOS exercises while work- to students. Likewise, struggling with semantic errors means that
ing on other tasks? while students can type syntax correctly, they fail to combine syntax
RQ3 Is there a correlation between completion of typ- into semantically valid statements. If students process more exam-
ing exercises and build failures in other work? ples of semantically valid statements from “canned” code, they will
be able to replicate those statements when applying CS concepts.
2 BACKGROUND Our typing exercises were written to focus on the application of a
CS concept (i.e. using a Finite State Machine design pattern). This
2.1 Decomposition & Subgoals design allows students to gain exposure to more canned code. If this
Program comprehension and subgoal generation are integral skills code reduces the debugging time for novices, then the addition of
in Computer Science [27]. To solve a problem, it is necessary to first this practice should improve students’ ability to write code more
understand the problem criteria and decompose that criteria into lists effectively, allowing for a better understanding of Computer Science.

106
Typing Exercises as Interactive Worked Examples for Deliberate Practice in CS Courses ACE’20, February 3–7, 2020, Melbourne, VIC, Australia

2.3 Expertise and Using Templates

Mastery requires learners to be exposed to a particular topic and be
given a chance to strengthen their knowledge of the topic through
practice. Ericsson defines expertise as “the result of extensive en-
gagement in relevant practice activities” [18], though biological and
sociological factors can also influence it [25]. In areas where the in-
dividual struggles or performs inadequately, they can use deliberate
practice as an intervention method for improvement. Ideally, this
should be spaced practice [20]. Spacing practice out forces schema
to be recalled from long-term memory and improves its durability.
Spacing out this deliberate practice thus helps build students’ under-
standing while rectifying any issues they may have had when they
were first introduced to the topic.
Drilling techniques are used in many technical skills, such as cook-
ing, music [18], sports [19], chess [11], and even medical diagnosis
[16]. Though Computer Science is more than programming, it is
often one of the first skills taught to novices. However, simply pro-
viding students with additional programming exercises as deliberate
practice may not improve the areas where students are struggling.
For example, syntax errors are some of the first problems experienced
Figure 1: A worked example of implementing Monte Carlo simulation to by novices [2] and providing more programming exercises will only
calculate the probability of rolling two 6’s with dice.
require them to spend more time repeating these errors. Students
without the proper technical literacy to overcome these errors will
fall behind and stay behind [1] as new CS concepts, their implemen-
tations, and new programming assignments are introduced. Thus
it may be beneficial to focus on improving novices’ technical liter-
2.2 Worked Examples
acy rather of continuing to have them struggle with higher-level
While the term “canned code” is not clearly defined in Brooks’ work, concepts (further falling behind). If typing exercises are used as an
they are analogous to worked examples. A worked example is a full or intervention tool, students get more opportunities to practice the syn-
partial solution to a problem for the purpose of instruction [3, 9, 51]. tax as a separate exercise supplemental to traditional programming
For novices, referring to worked examples can be more effective and exercises. In doing so, they will be able to move past the initial hurdle
efficient for problem solving [51, 54]. Additionally, comparing differ- of resolving syntax errors and focus on applying the covered concept.
ing solutions to a single problem fosters a more abstract understand- Ericsson also indicated that early specialization was an important
ing of the concept in many different STEM domains [12, 24, 41, 57]. factor for developing expertise. However, not every student major-
In CS courses, concepts are often illustrated through full or partial ing in Computer Science enters CS1 with this prior knowledge [22]
implementations of source code. This is less effective because exam- and women in particular may not have the same opportunities for
ple code is often in text form, allowing students to download or copy early specialization as males [21, 26, 32]. This matter is further com-
and paste it without fully understanding the code or which parts are pounded with recent increases in college enrollment numbers for
relevant to their problem. Computer Science [44]. With limited resources, CS departments have
Passively observing examples can be addressed in part by adding begun to increase requirements for enrollment in their programs
a level of interaction to the example. The ICAP framework in partic- [43]. These increases often require some level of prior knowledge
ular outlines different types of activities based on their user engage- in CS. However, a student without this prior knowledge should not
ment (interactive, constructive, active, and passive, respectively) be punished or penalized for not having the same skill level as their
[13]. By increasing the level of engagement, the learner engages in colleagues and the introductory course should be their gateway to-
information seeking and must process the example at a finer detail ward becoming familiar with Computer Science, no different than
than passively observing it [4, 35]. In the ICAP framework active an introductory psychology or foreign language course.
learning activities include “copying solution steps”, while passive Retyping source code allows novices to gain familiarity of the syn-
activities include “reading entire text passages silently/aloud with- tactical structure of a text-based programming language and, as dis-
out doing anything else”. Typing exercises, in turn, provide an active cussed in the following subsection, may be one of the methods prac-
learning opportunity where students can complete additional prac- ticed by prior novices when gaining early specialization. Soloway de-
tice writing code while being exposed to higher-level CS concepts. scribed programming knowledge as a mental set of “recurring basic
While constructive learning activities, such as self-explanation, have plans” that can be retrieved from long term memory when a problem
been shown to be useful, the benefits are typically limited to more similar to their mental model is presented [48]. If early specialization
knowledgeable students. Novices with a poor mental model may in CS is simply a result of building these plans at a younger age, then
overestimate their understanding of examples and refrain from do- novices without those plans should be given practice to construct
ing self-explanation or generate explanations that are not helpful them. In lieu of resolving a problem, CS students can recall an imple-
for learning[3, 24]. We are focusing on retyping worked examples mentation they’ve done previously and modify it to fit a new prob-
with the goal of helping beginner students move beyond these initial lem’s requirements. Thus, completing these typing exercises allow
limitations and to scaffold their learning processes so that they can
benefit from these constructive learning activities.

107
ACE’20, February 3–7, 2020, Melbourne, VIC, Australia Adam M. Gaweda, Collin F. Lynch, Nathan Seamon, Gabriel Silva de Oliveira, Alay Deliwa

students to practice resolving syntax errors and also produce learning 3 TYPOS EXERCISE DELIVERY PLATFORM
artifacts that may be referred to while learning future CS concepts. To measure the effectiveness of typing exercises, we developed a
platform known as TYPOS (named for how common syntax errors
are in programming). TYPOS is a Computer Science exercise delivery
2.4 Typing Exercises
platform where students are presented with various novel learning
While this work is novel, requiring novices to retype source code activities. These exercises range in complexity and further discussion
is not a wholly new concept. Prior to the widespread use of the In- about the platform can be found on the ARG Lab website1 . While we
ternet, children’s educational books on computers were often filled intend to study the relation between these exercises in future work,
with computer games; however, in order to play the game, the child we focus on typing exercises for this study to help understand their
needed to retype the printed syntax into their computer terminal [53]. benefit while learning CS.
Though retyping code was never explicitly used for instructional pur- Our typing exercises present example code as an image instead of
poses, children who did retype the code would inevitably get exposed in a text format. The image is presented beside an input box where
to syntax errors and debugging them. This early exposure to retyp- students retype the source code. The input box also highlights syntax
ing code may hold qualities that benefit learning Computer Science. similar to the image. Once a student submits the retyped code, it is
Recently, the series ‘Learn Python the Hard Way’ explicitly directs compared character by character to a copy of the source code stored
readers to “Type in each file exactly.” and that if they use copy and in our database. These exercises also included instructor comments
paste they “might as well not even do them” [46]. If the student does explaining various aspects of the syntax and pointing out minor
not enter the syntax correctly, the only error they can receive is about concepts, such as using special characters for string formatting (\t,
their precision. This is unlike a traditional programming exercise, \n), which may not be directly covered during lecture. While not
where syntax errors can be produced simultaneously to logic errors. requiring the same level of precision as programming code, the com-
There has been recent research into the effectiveness of typing ments included in the exercises were also measured for accuracy. If
exercises [30]. In Leinonen’s study, students were randomly pre- a discrepancy (or typo) appears between the student’s and the plat-
sented with optional typing exercises during the first two weeks form’s versions, the platform will highlight the first mismatching
of a 7-week semester. Students would receive immediate feedback line of code as shown in Figure 2. Students may continue modifying
from their tool if they entered an incorrect character. Their analysis their code until their submission matches the example.
found that students that completed the exercises made the same
number of errors as students that were not exposed to the exercises
and found no statistical significance to typing exercises improving
course grades. Our works differ such that our study spanned the en-
tirety of three 16-week semesters and that feedback was only given Figure 2: If a student submits code with typos, the platform will indicate
upon submission (rather than immediately after making a mistake). where the error occurs. In this example, the written code uses commas instead
We argue our approach more closely aligns with how students write of semicolons to separate expressions.
syntax, as they may not be aware of their typos until compilation.
Likewise, since CS courses commonly build on previously- covered Using an image of source code prevents the student from copying
material, typing exercises may hold more benefit over continuous the code and pasting into the input box. While these restrictions
practice, rather than only at the start of the semester. would not stop a dedicated cheater, the true audience of these exer-
Typing exercises are novel exercises in a similar vein to Parson’s cises are individuals who do not already have that level of expertise.
Puzzles [15, 31, 36]. Denny found notable correlations between Par- Image copies of the source code used in our exercises are found on
son’s Puzzle scores and exam code writing scores and Lopez con- our website2 .
cluded their analysis on Parson’s Puzzles by stating they may be
a lower skill than code tracing. The continued research and use of 4 STUDY
Parson’s Puzzles support our goals to study the impact on novel 4.1 Design
exercise types that could be practiced by students while learning
CS. Typing exercises, like Parson’s Puzzles or other novel exercises, We incorporated TYPOS into three semesters of an existing CS course
do not take as much time as programming exercises and can serve as a series of optional exercises. This course focuses predominantly
as a milestone for work completed in a given week. By splitting the on object-oriented programming and basic linear data structures.
activities of learning technical literacy and problem solving, this The sections were led by the same instructor, used the same materials
study shows that spaced deliberate practice of repeating worked and pacing, and used isomorphic assignments (to ensure academic
examples is a method to bolster students learning Computer Science. integrity across semesters). The students were not compensated for
In this study, we assert that typing exercises are a beneficial ac- their participation nor were they given course credit. Thus consent-
tivity that should be completed in addition to traditional coding ing students were self-selected to a group that wished to explore
exercises in CS courses. Since syntax errors are a result of impreci- additional content and improve their class performance. Further,
sion while writing code, these exercises form a deliberate practice in students could consent over whether to have their interactions used
resolving this issue quickly. Likewise, if each concept in CS builds on for data analysis while registering on the platform.
each other, then this spaced practice would reinforce foundational The course’s assessment plan includes weekly programming ex-
concepts while introducing new ones. These exercises would also ercises, two midterms,and a final exam. Students were also expected
produce artifacts that students can attach to Soloway’s recurring to complete 2-3 large-scale programming projects with increasing
basic plans, allowing them to reuse this code, which would allow levels of collaboration.The first project required students to complete
students to focus on problem-solving and subgoal generation, rather 1 https://research.csc.ncsu.edu/arglab/projects/typos/
than resolving syntax errors. 2 https://research.csc.ncsu.edu/arglab/projects/typos/exercises.html

108
Typing Exercises as Interactive Worked Examples for Deliberate Practice in CS Courses ACE’20, February 3–7, 2020, Melbourne, VIC, Australia

the implementation alone and the final project required groups of 2 Active students accounted for 75% of all activity on the platform
or 3 students. Students would store and collaborate their implementa- (5,556 views, 7,160 attempts, and 2,296 completions). We classified
tions using GitHub. Their implementations would be automatically these students into three categories based upon their overall ac-
tested across teaching staff unit test cases using the Jenkins con- tivity within the system: regular viewers that viewed exercises but
tinuous integration system. The majority of repositories had more attempted less than 30 over the course of the semester (17), regular
than 50 commits from the students, each of which provided a snap- attempters that attempted exercises but completed less than 30 over
shot of their implementation. Jenkins provided the students with the course of the semester (26), and regular completers who com-
three levels of feedback: a “red ball” indicates a “build failure” where pleted 30 or more exercises (34). The grade distributions for each
the student’s implementation could not be compiled, a “yellow ball” classification are shown in Table 2.
which indicates that the code compiled but failed code coverage or
Table 2: Final Grade Distribution across Categories
a unit test, and a “green ball” to indicate it compiled and passed all
code coverage and unit tests. Students used the Eclipse development Category Total A B C D F
environment for the assignments which were graded based on style, Viewers 17 9 4 3 - 1
documentation, and design strategy. Attempters 26 11 11 2 - 2
At the beginning of the semester, students were introduced to Completers 34 19 14 1 - -
TYPOS and the participating students were required to register with
the platform. The link to joining the platform was included at the In the following subsections, we analyze student performance
top of the course’s web page and students could register at any time in the course compared to their usage of the platform and their
during the course of the semester. interactions with the optional exercises compared to their graded
The course consisted of 24 lectures, 2 midterm exams, and a final programming projects.
exam. A module was unlocked each lecture, containing one to five
exercises pertaining to that day’s lecture topic; the instructor would 4.2 Student Performance Analysis
inform the class about them before each lecture. Students were not We first started with simple descriptive statistics and RQ1 to see how
required to complete the exercises in a particular order and could students used typing exercises and any effects isolating syntax errors
repeat exercises as often as they chose. Over the semester, a total of had on student performance. Table 3 shows the regular completers
66 exercises were provided, averaging 33 lines of code each. Students made over 10 times as many attempts on the exercises as viewers and
spent on average 400 seconds (~7 minutes) completing each exercise almost three times as many as attempters. These observations are
. Some of the most viewed exercises include implementing finite consistent with Ahadi et al.’s findings that one of the most common
state machines using a “while-switch” design and inheritance, two initial issues students face comes from syntax errors [2]. Completing
of the more difficult topics in the course and an important portion a simple typing exercise still caused students to make typos in source
of the second midterm and final exam. A break down of the ten most code and were not resolved after a single correction.
completed exercises across all semesters can be found in Table 1.
Table 3: Average Exercise Activity across Categories
Table 1: Most Frequently Completed Typing Exercises (* first module exercise) Category Viewed Attempted Completed
Title Completed Viewed Viewers 58.12 13.71 3.35
Point Class* 202 540 Attempters 60.15 56.15 18.00
Using the Point Class* 143 248 Completers 88.35 160.79 52.09
Intro to JUnit 130 237
ArrayLists 123 228
NewsStory Class 116 177
Figure 3: Average Exam Grades for Each Group
While-Switch FSMs 116 529
Null Dereferencing 107 197
NewsFlag Parent Class 102 176
The Singleton Pattern 86 197
Code Coverage (with Tests) 83 133

In total, 337 students made 7,334 views, 10,097 incorrect attempts,

and 3,024 completions on the platform. Viewing an exercise required
the student to select it within a module and then view the code im-
age. Attempts are considered submissions that were incorrect, and
completions are submissions that were correct.
The majority (77%) of students did not use TYPOS regularly, 235
students logged in infrequently while 25 never logged in at all. Based
on our exploratory analysis, the infrequent students only viewed Our student activity data was normally distributed and thus we
exercises that they were explicitly told would be on the exams and used an ANOVA single factor analysis to compare our categories
their overall use of the platform was sporadic and minimal. We omit- against students’ final course grades. Figure 3 shows that all students
ted these students from further analysis because it was impossible performed relatively the same during the first midterm. Based on
to determine the effect of isolating syntax error practice on their Table 3, attempters made as many attempts as views. We interpret
success in the course. We opted instead to examine how active users this to mean attempters retyped the exercise, but upon an incorrect
(N=77) differed from each other. submission, could not find the error or copied their code into their

109
ACE’20, February 3–7, 2020, Melbourne, VIC, Australia Adam M. Gaweda, Collin F. Lynch, Nathan Seamon, Gabriel Silva de Oliveira, Alay Deliwa

IDE, which would correct their mistakes for them. Thus while at- alternated between the activities. In fact, TYPOS usage was almost
tempters did get practice retyping code, they did not get practice entirely done prior to the lecture. We believe that students, waiting
identifying and rectifying syntax errors. for the lecture to begin, chose to complete these exercises instead
of sitting idle. We ascribe this to the fact that these exercises were
Table 4: P-values for all and lower-performing completers across course grade smaller in scope, took less time to complete, and could be completed
and learning gains
before class, unlike their programming assignments. Our results mir-
Course Grades Exam Learning Gains
ror those in Sheshadri’s results, which found students would interact
Active Completers 0.065 0.161
primarily with a single learning environment, seldom interleaving
Initial Low-Performers 0.038 0.041
types of practice [47]. Regardless of reason, students did not visit
TYPOS while working on their other course assignments. It is pos-
As the semester progressed, regular completers earned higher sible, however, that since students could download their completed
exam grades and final course grades when compared to other regular submissions, they may have chosen to do this and then refer to their
users (see Table 4). One of our primary goals is to study whether lower learning artifacts while code writing rather than visiting TYPOS
performing students would benefit from practicing simple founda- modules directly.
tional exercises. We sampled students that scored below the median
on the first midterm. Initially lower-performing students comprised 4.4 Assignment Implementation Analysis
roughly 47% of our active user population, 16 were completers, 11
Our final research question assesses whether students who prac-
were attempters, and 9 were viewers (N=36). Similar to the total pop-
ticed typing exercises would be more aware of mistakes while code
ulation, low-performing completers earned higher course grades and
writing and thus avoid or correct them sooner. Leinonen et al. found
also made the highest learning gains between the first and final exam
that typing practice did not impact the number of mistakes students
among groups. Likewise, all initially lower-performing students that
made; however, their study only observed the first two weeks of
used the platform had higher than average learning gains.
the semester [30]. We believe that more repetition observing syntax
However, these exams were presented on paper and included as-
mistakes over the course of the semester would reduce the number of
sessments besides code writing. Thus exam scores are not a strong
compilation errors made by students. Each time a student committed
indicator for the effect of typing exercises since students were as-
their implementation to GitHub, Jenkins would download their code
sessed on more than their code writing ability. The first midterm
and test it. At the conclusion of each semester, we collected each
scores were relatively low across all groups and thus some students
repository’s compilation data and compared the number of build
may have simply been more motivated to improve. While our popula-
failures with TYPOS usage.
tion was self-selected, these students may have improved regardless
We found that all regular TYPOS usage was negatively correlated
of their interactions with TYPOS. We do note that some of the most
to the number of build failures a particular repository would have (see
commonly practiced TYPOS exercises were focused on concepts that
Table 5). This means that the more exercises completed, attempted,
were covered on the exams. Thus for many students it may be the
or even viewed would reduce the number of build failures students
case that they used the platform for exam preparation rather than as
experienced. Note, the terms view, attempt, and complete refer to
a practice tool. However, if retyping code examples was part of their
the number of activities made by a student on TYPOS rather than the
preparation, this would open discussions on how students should
labels used in the previous analysis. This is because we wanted to
best prepare for CS exams. In the following section, we made efforts
measure if either of the three behaviors had any relation to students’
to assess other factors, such as platform usage and build failures in
code writing ability.
programming projects, to account for this limitation.
We do note that these correlations, while significant, are low. As
4.3 Usage Analysis we describe in our Limitations section, this could be attributed to
the students being self-selected. However, if typing exercises take
Following the overall effect of typing exercises on performance, we less time than programming exercises, these correlations could help
moved our focus to RQ2. Students will frequently refer to course reduce the time students spend working on their implementations.
materials and potentially third-party websites, such as StackOver-
flow, while completing traditional programming exercises. While
Table 5: Pearson’s r Comparing TYPOS Activity to Build Failures (* p<0.05, **
we do not condone students simply “copy and paste” from these p < 0.01)
websites, this behavior indicates that students are reviewing coding # Viewed # Attempted # Completed
examples while working on their implementations. Therefore, we Active Students -0.045** -0.048** -0.110**
wanted to see if students were exhibiting similar behavior with the Initial Low-Performers -0.132* -0.050** -0.144**
coding examples on TYPOS.
We extracted the feedback Jenkins provided students to assess
whether students would reference TYPOS examples while working We chose to use the same cohort of active students to measure the
on their own code assignments. We looked at times that students relation between their TYPOS usage and the number of compilation
made activities on TYPOS vs. Jenkins. We segmented student in- errors, or build failures, they received while developing their course
teractions into sessions similar to Sheshadri et al. [47]. Their study projects. We found that completing TYPOS exercises and build fail-
used n-gram frequencies to see how students interacted with var- ures were negatively correlated. Viewing and attempting exercises
ious learning environments. We hypothesized that during sessions were also negatively correlated, but to a much lesser degree. These
focused on implementing project solutions, students would visit findings suggest that simply providing worked examples may not be
TYPOS and reference the examples. enough and that providing some degree of engagement should be in-
Contrary to our assumptions, students tended to either complete cluded. This is consistent with the ICAP framework which notes that
TYPOS exercises or their course projects during a session, but rarely active learning activities, such as “copying solution steps", benefit

110
Typing Exercises as Interactive Worked Examples for Deliberate Practice in CS Courses ACE’20, February 3–7, 2020, Melbourne, VIC, Australia

learning more than passive activities, such as "reading entire text pas- as students become more familiar with syntax. We will evaluate this
sages silently” [13]. Interestingly, the Pearson’s correlations in Table in future studies.
5 show attempting and viewing exercises were similar for active stu- Finally, the typing exercises were also designed to mirror the top-
dents, rather than with completing exercises. This could suggest that ics covered in a single college-level CS course. It is therefore not clear
the benefit of typing exercises is not in retyping syntax, but rather how the platform would work with students at different grade levels,
in experiencing typos, assessing their location, and correcting them. as a graded requirement for a course, or for those who are using
As with our performance analysis, we isolated the low-performing it outside of a course. We intend to extend our analysis to include
students and wanted to see how or if these correlations changed. For deliberate practice across other novel exercise types and how bests
these students, both viewing and completing exercises were nega- to present them to students.
tively correlated to the number of build failures. We interpret this to
mean that while all active students benefited from typing exercises,
initially low-performing students benefited from simply having the
additional examples available to them. Though our correlations are
not large, they do show some benefits to students that engaged with
the additional exercises. Lower performing students benefited from 6 CONCLUSIONS
simply having more examples to reference and all students benefited In order to become effective in any domain, novices should engage in
from retyping the example. deliberate practice [18, 38] and devote time to practicing and refining
their understanding of both high-level concepts and low-level skills.
In this work, we found that self-selected students who completed
typing exercises performed better than students that did not, though
5 LIMITATIONS with marginal significance. By focusing on core syntax, these stu-
One limitation of this work is that students were self-selected to dents performed deliberate practices on one of the first pitfalls faced
complete these exercises. The active users in our study earned rel- by many CS students. With more exposure to these types of errors, it
atively low scores on their first exam. These grades may have been is likely they began to identify them sooner and resolved them before
enough motivation for these students to improve their performance compilation. As a result, they resolved syntax errors sooner, and
in the course. Though active users of TYPOS did earn higher grades thus focused on more cognitively demanding tasks, such as planning
overall, this correlation was only marginally significant, and we solutions to programming exercises. This was further supported by
were not able to conduct additional surveys to determine whether students that completed typing exercises showing fewer instances
these students also sought out other methods for improvement such of build failures while completing graded assignments.
as attending office hours or other online materials. Typing exercises can also serve as an explicit guide to implement-
Another limitation is that the students were required to complete ing core concepts without risking academic integrity violations.
the exercises on a web platform. As mentioned earlier, the low com- Stack Overflow is arguably one of the most well known websites
pletion of exercises by viewers may be because they elected to retype among current Computer Science students and even describes it-
the exercises within their own IDEs. Thus they may not be pure self as a ‘question and answer site for professional and enthusiast
‘viewers’, but unrecorded attempters or completers. Attempters may programmers’ [49]. While many code support sites have policies to
have also started using the platform, but moved their code to an IDE prevent plagiarism [14], students still use them to find example code
that could automatically detect errors. This could also be the same and are often tempted to incorporate it into their own work. Though
reason attempting typing exercises were not strongly correlated we cannot say retyping code will stop students from copying and
with build failures. Likewise, we determined the thresholds for each pasting habits, we are providing students with learning artifacts that
category. These thresholds may not be constant and could require ad- they have created. Thus, since they already have an implementation
justing, thus invalidating our model. Therefore, it may make sense to they made themselves, they may be less inclined to resort to using
provide in-IDE access to the system or to use a clustering algorithm code found on third-party websites.
to separate students. We encourage CS instructors to include typing exercises as a part
One additional limitation that was brought to our attention by of their current course designs. These exercises do not require a large
prior discussion at SIGCSE is the lack of support for students with portion of students’ time and have clearly defined goals for students
visual impairments or other limitations [5, 45, 50]. As structured the to accomplish. These exercises provide students with syntactical
current exercises do not provide necessary accommodations. How- practice; however, they should compliment, not overshadow tradi-
ever we believe that standard approaches such as audio recordings of tional assessments. If these exercises are included as a graded activity,
syntax, or other well-studied human-factors accommodations, could we suggest they be weighted similar to activities such as attendance
be used to provide the same cognitive and pedagogical benefits to or in-lecture exercises. The exercises could even replace traditional
students with impairments. Thus, we believe that we can draw from attendance, replacing the passive action of "simply showing up" with
existing HCI work to support students with other exercise types that a more active approach. This would ensure students are not only
still provide similar benefits. exposed to the concept, but also given an opportunity to refine their
Moreover, as noted by Sweller, "it cannot be assumed that all technical ability to implement them.
worked examples under all circumstances will have beneficial conse- While typing exercises are only a small subset of exercises that
quences" [52]. Since we were only able to show marginal significance novice CS students can complete as they progress to mastery, they
and small correlations, this skill may not be as beneficial as other provide students with repetitions in low-level debugging tasks that
exercise types. Altadmri argued that the frequency of syntax errors can inhibit higher level progress. Further, they provide them with
made by novices declined over time [2]. Thus, while typing exercises learning artifacts that they can use to support higher-level problem
may provide initial benefits, their overall effectiveness may decline solving.

111
ACE’20, February 3–7, 2020, Melbourne, VIC, Australia Adam M. Gaweda, Collin F. Lynch, Nathan Seamon, Gabriel Silva de Oliveira, Alay Deliwa

7 FUTURE WORK [18] K. A. Ericsson, R. Krampe, and C. Tesch-romer. The role of deliberate practice in
the acquisition of expert performance. Psychological Review, pages 363–406, 1993.
Our results open a new discussion on the type of practice students [19] K. A. Ericsson, M. J. Prietula, and E. T. Cokely. The making of an expert. Harvard
should be doing to improve in Computer Science. However, students business review, 85(7/8):114, 2007.
[20] Erwin Oliver Finkenbinder. The curve of forgetting. The American Journal of
who can successfully complete typing exercises may still not be Psychology, 24(1):8–32, 1913.
ready to complete traditional programming exercises. In our future [21] Allan Fisher, Jane Margolis, and Faye Miller. Undergraduate women in computer
work, we will investigate the relation that these exercises have to science: experience, motivation and culture. In ACM SIGCSE Bulletin, volume 29,
pages 106–110. ACM, 1997.
other learning activities, such as Parson’s Puzzles, output prediction, [22] A. Funke, M. Berges, and P. Hubwieser. Different perceptions of computer science.
bug localization, and self-explanation. Rather than comparing activ- In 2016 International Conference on Learning and Teaching in Computing and
ities against each other, we plan to analyze the sequence in which Engineering (LaTICE), pages 14–18, March 2016.
[23] Adam M Gaweda and Collin F Lynch. Giving students canned code using typing
students complete these exercises. Since each of these activities are exercises. In Proceedings of the 50th ACM Technical Symposium on Computer
beneficial to novices while learning CS concepts, our focus is to Science Education, pages 1281–1281. ACM, 2019.
[24] Peter Gerjets, Katharina Scheiter, and Richard Catrambone. Can learning from
assess when a particular exercise is most appropriate. molar and modular worked examples be enhanced by providing instructional
These activity sequences will be beneficial to both instructors explanations and prompting self-explanations? Learning and Instruction,
and designers of intelligent tutoring systems. Activity sequences 16(2):104–121, 2006.
[25] D. Hambrick, F. Oswald, E. Altmann, E. Meinz, F. Gobet, and G. Campitelli.
that identify initially struggling students who ultimately master a Deliberate practice: Is that all it takes to become an expert? Intelligence, 45:34
CS concept are of particular interest. Their activity sequence could – 45, 2014. Acquiring Expertise: Ability, Practice, and Other Influences.
be used as an intervention method when an future at-risk student [26] Steven Henn. Why women stopped coding, 2014.
[27] Cindy E Hmelo-Silver. Problem-based learning: What and how do students learn?
is identified. As activity sequencing becomes more well-defined, Educational psychology review, 16(3):235–266, 2004.
instructors and intelligent tutoring systems will be able to identify [28] Robert Kowalski. Computational logic and human thinking: how to be artificially
intelligent. Cambridge University Press, 2011.
these students and provide them with a tailored list of practice prob- [29] Essi Lahtinen, Kirsti Ala-Mutka, and Hannu-Matti Järvinen. A study of the
lems based on previously successful behaviors so that they can catch difficulties of novice programmers. In Acm Sigcse Bulletin, volume 37, pages 14–18.
up, rather than fall further behind. ACM, 2005.
[30] Antti Leinonen, Henrik Nygren, Nea Pirttinen, Arto Hellas, and Juho Leinonen.
Exploring the applicability of simple syntax writing practice for learning
REFERENCES programming. In Proceedings of the 50th ACM Technical Symposium on Computer
[1] Alireza Ahadi, Raymond Lister, and Donna Teague. Falling behind early and Science Education, pages 84–90. ACM, 2019.
staying behind when learning to program. In Proceedings of the 25th Psychology [31] Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. Relationships
of Programming Conference, PPIG, volume 14, 2014. between reading, tracing and writing skills in introductory programming. In
[2] A. Altadmri and N. Brown. 37 million compilations: Investigating novice Proceedings of the fourth international workshop on computing education research,
programming mistakes in large-scale student data. In Proceedings of the 46th ACM pages 101–112. ACM, 2008.
Technical Symposium on Computer Science Education, SIGCSE ’15, pages 522–527, [32] Jane Margolis and Allan Fisher. Unlocking the clubhouse: Women in computing.
New York, NY, USA, 2015. ACM. MIT press, 2003.
[3] Robert K Atkinson, Sharon J Derry, Alexander Renkl, and Donald Wortham. [33] Bruce M McLaren and Seiji Isotani. When is it best to learn with all worked
Learning from examples: Instructional principles from the worked examples examples? In International Conference on Artificial Intelligence in Education, pages
research. Review of educational research, 70(2):181–214, 2000. 222–229. Springer, 2011.
[4] Robert K Atkinson and Alexander Renkl. Interactive example-based learning [34] Nicholas Metropolis and Stanislaw Ulam. The monte carlo method. Journal of
environments: Using interactive elements to encourage effective processing of the American statistical association, 44(247):335–341, 1949.
worked examples. Educational Psychology Review, 19(3):375–386, 2007. [35] Roxana Moreno and Richard Mayer. Interactive multimodal learning environ-
[5] Catherine M Baker, Cynthia L Bennett, and Richard E Ladner. Educational ments. Educational psychology review, 19(3):309–326, 2007.
experiences of blind programmers. In Proceedings of the 50th ACM Technical [36] Dale Parsons and Patricia Haden. Parson’s programming puzzles: A fun and
Symposium on Computer Science Education, pages 759–765. ACM, 2019. effective learning tool for first programming courses. In Proceedings of the 8th
[6] Joseph E Beck and Yue Gong. Wheel-spinning: Students who fail to master a skill. Australasian Conference on Computing Education - Volume 52, ACE ’06, pages 157–
In International Conference on Artificial Intelligence in Education, pages 431–440. 163, Darlinghurst, Australia, Australia, 2006. Australian Computer Society, Inc.
Springer, 2013. [37] E. Patitsas, J. Berlin, M. Craig, and S. Easterbrook. Evidence That Computer
[7] Marc Berges and Peter Hubwieser. Evaluation of source code with item response Science Grades Are Not Bimodal. In Proceedings of the 2016 ACM Conference on
theory. In Proceedings of the 2015 ACM Conference on Innovation and Technology International Computing Education Research, ICER ’16, pages 113–121, New York,
in Computer Science Education, pages 51–56. ACM, 2015. NY, USA, 2016. ACM.
[8] Jeffrey Bonar and Elliot Soloway. Uncovering principles of novice programming. [38] M. Poppendieck. Deliberate practice in software development. a, 2009.
In Proceedings of the 10th ACM SIGACT-SIGPLAN symposium on Principles of [39] Scott R Portnoff. The introductory computer programming course is first and
programming languages, pages 10–13. ACM, 1983. foremost a language course. ACM Inroads, 9(2):34–52, 2018.
[9] Joel Brandt, Mira Dontcheva, Marcos Weskamp, and Scott R Klemmer. Example- [40] D. Pritchard. Frequency distribution of error messages. In Proceedings of the
centric programming: integrating web search into the development environment. 6th Workshop on Evaluation and Usability of Programming Languages and Tools,
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, PLATEAU 2015, pages 1–8, New York, NY, USA, 2015. ACM.
pages 513–522. ACM, 2010. [41] Bethany Rittle-Johnson and Jon R Star. Does comparing solution methods
[10] Ruven Brooks. Towards a theory of the cognitive processes in computer facilitate conceptual and procedural knowledge? an experimental study on
programming. International Journal of Man-Machine Studies, 9(6):737–751, 1977. learning to solve equations. Journal of Educational Psychology, 99(3):561, 2007.
[11] N. Charness, M. Tuffiash, R. Krampe, E. Reingold, and E. Vasyukova. The role of [42] Kelly Rivers, Erik Harpstead, and Kenneth R Koedinger. Learning curve analysis
deliberate practice in chess expertise. Applied Cognitive Psychology, 19(2):151–165, for programming: Which concepts do students struggle with? In ICER, pages
2005. 143–151, 2016.
[12] Michelene TH Chi, Paul J Feltovich, and Robert Glaser. Categorization and [43] Eric Roberts. A history of capacity challenges in computer science, 2016.
representation of physics problems by experts and novices. Cognitive science, [44] J. F. Sargent Jr. The U.S. Science and Engineering Workforce: Recent, Current,
5(2):121–152, 1981. and Projected Employment, Wages, and Unemployment, 2017.
[13] Michelene TH Chi and Ruth Wylie. The icap framework: Linking cognitive engage- [45] Emmanuel Schanzer, Sina Bahram, and Shriram Krishnamurthi. Accessible
ment to active learning outcomes. Educational psychologist, 49(4):219–243, 2014. ast-based programming for visually-impaired programmers. In Proceedings of
[14] Joel Coehoorn. How to Ask for Help - Stack Overflow - the 50th ACM Technical Symposium on Computer Science Education, pages 773–779.
https://meta.stackoverflow.com/a/334823/1558159, 2017. ACM, 2019.
[15] Paul Denny, Andrew Luxton-Reilly, and Beth Simon. Evaluating a new exam [46] Zed A Shaw. Learn Python the hard way: A very simple introduction to the
question: Parsons problems. In Proceedings of the fourth international workshop terrifyingly beautiful world of computers and code. Addison-Wesley, 2013.
on computing education research, pages 113–124. ACM, 2008. [47] Adithya Sheshadri, Niki Gitinabard, Collin F Lynch, Tiffany Barnes, and Sarah
[16] R. Duvivier, J. van Dalen, A. Muijtjens, V. Moulaert, C. van der Vleuten, and Heckman. Predicting student performance based on online study habits: a study
A. Scherpbier. The role of deliberate practice in the acquisition of clinical skills. of blended courses. arXiv preprint arXiv:1904.07331, 2019.
BMC Medical Education, 11(1):101, 2011. [48] Elliot Soloway and Kate Ehrlich. Empirical studies of programming knowledge.
[17] Carol S Dweck. Mindset: The new psychology of success. Random House IEEE Transactions on software engineering, (5):595–609, 1984.
Incorporated, 2006. [49] Joel Spolsky and Jeff Atwood. Stack Overflow - https://www.stackoverflow.com.

112
 Typing Exercises as Interactive Worked Examples for Deliberate Practice in CS Courses ACE’20, February 3–7, 2020, Melbourne, VIC, Australia

[50] Andreas Stefik, Richard E Ladner, William Allee, and Sean Mealin. Computer review, 17(2):147–177, 2005.
science principles for teachers of blind and visually impaired students. In [55] Susan Wiedenbeck, Vikki Fix, and Jean Scholtz. Characteristics of the mental rep-
Proceedings of the 50th ACM Technical Symposium on Computer Science Education, resentations of novice and expert programmers: an empirical study. International
pages 766–772. ACM, 2019. Journal of Man-Machine Studies, 39(5):793–812, 1993.
[51] John Sweller. Cognitive load during problem solving: Effects on learning. [56] Edward A Youngs. Human errors in programming. International Journal of
Cognitive science, 12(2):257–285, 1988. Man-Machine Studies, 6(3):361–376, 1974.
[52] John Sweller. Cognitive load theory, learning difficulty, and instructional design. [57] Rui Zhi, Thomas W. Price, Samiha Marwan, Alexandra Milliken, Tiffany Barnes,
Learning and instruction, 4(4):295–312, 1994. and Min Chi. Exploring the impact of worked examples in a novice programming
[53] Usbourne. Usborne 1980s Computer Books. environment. In In Proceedings of the 50th ACM Technical Symposium on Computer
[54] Jeroen JG Van Merrienboer and John Sweller. Cognitive load theory and complex Science Education, SIGCSE ’19, 2019.
learning: Recent developments and future directions. Educational psychology

113

View publication stats