Toward a Shared Understanding of Competency in Programming:

An Invitation to the BABELnot Project

Raymond Lister Malcolm Corney James Curran

Faculty of Engineering and Faculty of Science and Technology, School of Information Technologies,
Information Technology, Queensland University of University of Sydney,
University of Technology, Sydney, Technology, Sydney, NSW, Australia
Sydney, NSW, Australia Brisbane, Qld, Australia [email protected]
[email protected] [email protected]

Daryl D’Souza Colin Fidge Richard Gluga

School of Computer Science and Faculty of Science and Technology, School of Information Technologies,
Information Technology, Queensland University of University of Sydney,
RMIT University, Technology, Sydney, NSW, Australia
Melbourne, Vic, Australia Brisbane, Qld, Australia [email protected]
[email protected] [email protected]

Margaret Hamilton James Harland James Hogan

School of Computer Science and School of Computer Science and Faculty of Science and Technology,
Information Technology, Information Technology, Queensland University of
RMIT University, RMIT University, Technology,
Melbourne, Vic, Australia Melbourne, Vic, Australia Brisbane, Qld, Australia
[email protected] [email protected] [email protected]

Judy Kay Tara Murphy Mike Roggenkamp

School of Information Technologies, School of Information Technologies, Faculty of Science and Technology,
University of Sydney, University of Sydney, Queensland University of
Sydney, NSW, Australia Sydney, NSW, Australia Technology,
[email protected] [email protected] Brisbane, Qld, Australia
[email protected]

Judy Sheard Simon Donna Teague

Faculty of Information Technology, School of Design, Communication, Faculty of Science and Technology,
Monash University, Caulfield East, & IT, University of Newcastle, Queensland University of
Victoria, Australia Ourimbah, NSW, Australia Technology,
[email protected] [email protected] Brisbane, Qld, Australia
[email protected]

Abstract• learning goals associated with programming. It will also

The ICT degrees in most Australian universities have a be necessary to benchmark exam questions that are
sequence of up to three programming subjects, or units. mapped onto this framework. As part of the project,
BABELnot is an ALTC-funded project that will workshops are planned for ACE 2012, ICER 2012 and
document the academic standards associated with those ACE 2013, to elicit feedback from the broader
three subjects in the six participating universities and, if Australasian computing education community, and to
possible, at other universities. This will necessitate the disseminate the project’s findings. The purpose of this
development of a rich framework for describing the paper is to introduce the project to that broader
Australasian computing education community and to
invite their active participation.
Copyright © 2012, Australian Computer Society, Inc. This Keywords: programming, objectives, assessment.
paper appeared at the 14th Australasian Computing Education
Conference (ACE2012), Melbourne, Australia, January- 1 Introduction
February 2012. Conferences in Research and Practice in It is very common for ICT degrees to incorporate a
Information Technology, Vol. 123. M. de Raadt and A. sequence of up to three programming subjects (also
Carbone, Eds. Reproduction for academic, not-for-profit known as courses, papers, or units of study).
purposes permitted provided this text is included. Traditionally, these three subjects have formed part of the
compulsory ‘core’ of ICT degrees, particularly software PF/Fundamental Constructs [core]
engineering degrees. There is certainly a great deal of
variety between institutions as to what is covered in the Minimum core coverage time: 9 hours
subjects. While the first is typically thought of as an
introduction to programming, the second might be a Topics:
direct continuation of programming concepts, a data • Basic syntax and semantics of a higher-level
structures subject, a subject addressing program access to language
databases, and so on; and even more variation can be
expected in the third subject. Nevertheless, it appears to • Variables, types, expressions, and assignment
be the case that many ICT degrees identify three specific • Simple I/O
subjects as an effective programming stream, and it is • Conditional and iterative control structures
these three subjects with which this project is concerned. • Functions and parameter passing
Despite the centrality of these three programming
subjects, computing academics remain dissatisfied with • Structured decomposition
the effectiveness of these subjects. Many students are
also dissatisfied: in a widely discussed paper describing Learning Objectives:
the educational ‘Grand Challenges’ in computing, 1. Analyze and explain the behavior of simple programs
McGettrick et al (2004) note that: involving the fundamental programming constructs
covered by this unit.
“educators cite failure in introductory programming 2. Modify and expand short programs that use standard
courses and/or [student] disenchantment with conditional and iterative control structures and
programming as major factors underlying poor functions.
student retention”.
3. Design, implement, test, and debug a program that uses
In programming subjects, as with most Australian each of the following fundamental programming
university subjects, the semester begins in each constructs: basic computation, simple I/O, standard
classroom with the ritual distribution of the subject conditional and iterative structures, and the definition
outline, which provides a brief description of the subject, of functions.
the topics to be covered, and the assessment scheme. 4. Choose appropriate conditional and iteration constructs
Although such outlines often run to many pages, the for a given programming task.
document can be ambiguous. For example, consider the
following objective, taken from the outline of an 5. Apply the techniques of structured (functional)
introductory programming subject at one of the decomposition to break a program into smaller pieces.
universities participating in this project: 6. Describe the mechanics of parameter passing.
On successful completion of this subject, the student Figure 1: An extract from Computer Science
will be able to ... Demonstrate a working knowledge Curriculum 2008 (ACM/IEEE, 2008)
of the basic constructs in the object-oriented
language Java. subject varies over time. A change to the final exam is
one of the most important yet subtle ways that a subject
Which constructs are the “basic” constructs? What does can change. If one loiters long enough in a departmental
it mean to have a “working knowledge”, and how does a tea room around the time of semester when exams are
student “demonstrate” it? Figure 1 shows an extract from being written, one will hear quite passionate complaints
Computer Science Curriculum 2008 (ACM/IEEE, 2008), that Professor Bloggs has ‘watered down’ the final exam
which manifests numerous similar ambiguities. in a particular subject (e.g. by changing from free
Furthermore, while students may think of outlines as response to multiple choice). Changes to an exam often
the contract between them and their teacher, outlines are do not require changes to the subject outline or any other
conscripted into many roles. For example, outlines are documentation, and can thus be made with little
presented to professional accreditation committees as management oversight. Academics who teach
evidence that the required subject matter is being taught. downstream of that subject may not even be aware of the
Mappings are sometimes made from subject outlines to a change until well after it has taken place.
university’s graduate attributes. When a student moves to
a new university and seeks credit for prior study, outlines 1.1 The Relationship with Software
are used to establish subject equivalence between the two Engineering
universities. Very importantly, outlines are also a contract Software engineering as a discipline has wrestled with
between teachers. In a three-semester sequence of problems that are analogous to the pedagogical problems
programming subjects, for example, the second and third described above. A software engineering project usually
semester teachers rely upon the outline of the previous begins with a long negotiation between the software
subject to define what students should know at the start of developers and the various stakeholders. The negotiation
semester – and sometimes those teachers feel justified in culminates in a design document, often called a
complaining that the students cannot actually do what the specification, which forms a contract between the
previous subject’s outline says they can do. software developers and the various stakeholders. Even
If we ignore the relationship of a given subject to other the most comprehensive specification documents leave
subjects, be they different subjects at the same university implicit some aspects of the proposed system, which are
or equivalent subjects at other universities, even a given
remembered as shared understandings and oral In this project, learning outcomes will tend to be
agreements arising from certain meetings. After the articulated in terms of a characterisation of suitable
implementation of the software begins, changes are assessment tasks, for example,
inevitably made to the specification. Some changes are
“On successful completion of this subject, a
documented, while others remain implicit. Almost
passing student will be able to implement iterative
inevitably, not everyone is aware of, or happy with, some
algorithms on arrays, such as linear search,
of the changes. While software engineering is by no
binary search and quadratic sorting algorithms, in
means a solved problem, some aspects of the culture of
approximately half an hour, without reference to
software engineering can usefully be adopted to attack the
external notes”.
problems in ‘pedagogical engineering’ described above.
Thus academics with an ICT background bring a special Note that this is merely an illustrative example, not a
perspective to specifying academic standards. recommendation of a standard to be adopted. Like
Wright, Hadgraft, and Cameron (2010), it is not our
1.2 The BABELnot Project: Desired Outcomes intention to be prescriptive about what students at a
The above considerations led the authors to propose the particular institution should know, but rather to provide
BABELnot project. (See section 8 for an explanation of the framework within which academics at that institution
the name.) We successfully applied to the Australian might be prescriptive.
Learning and Teaching Council (ALTC) for funds to
support the project across the six participating 2 Background
institutions. The funded work of the project began in This section reviews relevant prior work that motivated
October 2011 and will continue until August 2013. the development of this project and influenced the
Our aim is to achieve consensus on a framework for project’s design.
describing learning outcomes in computer programming,
specifically the teaching of programming in the first three 2.1 The BRACElet Project
semesters, and also on how to map between learning A number of papers about the BRACElet project have
outcomes and exam questions. We understand that been presented at past ACE conferences (e.g. Whalley et
assessment in programming subjects is not restricted to al, 2006). Work on BRACElet started in New Zealand in
written exams, and that some learning outcomes are often 2004. In 2007, the ALTC funded a fellowship project by
assessed by way of other forms of assessment such as Lister and Edwards to explicitly extend BRACElet into
assignments and practical tests; but these other forms of Australia (Lister & Edwards, 2010). The final BRACElet
assessment are beyond the current scope of the project. workshop was held in 2010 (Clear et al, 2011).
Our desired outcomes are: BRACElet recruited academics from multiple
• The creation of a bottom-up, action research approach universities into an action research approach that
to articulating learning outcomes, in the context of the involved the systematic collection of evidence from end-
first three programming subjects of-semester programming exams. As part of this process
the project participants formulated ideas on where the
• A culture of scholarly teaching in ICT, spanning problems lay for novice programmers, devised exam
institutional boundaries, with a discourse based in questions to test these ideas, and collected and analysed
evidence rather than anecdote the data from the end-of-semester exams. This process
• Exams that are a more valid and reliable indicator of was repeated several times. Contrary to the intuitions of
student programming ability many computing academics, the project participants
• Better learning of programming by students found that students tend not to have problems with the
low level ‘nuts and bolts’ of programming. Instead they
• Attraction and retention of more students to have difficulties fitting the pieces together to see the
programming and software engineering larger picture − they ‘cannot see the forest for the trees’.
More specifically, our desired, measurable project Many traditional exam questions, however, largely test
deliverables are: the novice programmer on the lower level nuts and bolts,
• A system for describing learning outcomes and and learning outcomes are often expressed in terms of
assessment by written exam. A method for mapping these nuts and bolts.
between learning outcomes and exam assessment, Three workshops were held within Australia during
applicable to the first three programming subjects the funding period of the ALTC fellowship. A total of 21
• The learning outcomes of the first three programming Australian academics, from 14 different Australian
subjects, from at least the six participating universities, either attended these workshops or actively
universities, re-expressed within the system participated in the project electronically. Academics from
at least seven Australian universities have used end-of-
• A document summarising an archive of exam semester exam questions that were designed as part of
questions, with meta-tags mapping the questions to this project. The project has also attracted international
the system, serving as examples for use by other attention, with academics from 14 universities in seven
academics countries actively participating in data collection and
• Performance data from real students for a subset of analysis. During the ALTC Fellowship funding period, 26
the archived exam questions project participants (co-)authored 16 published papers,
further disseminating the outcomes of the project.
2.2 Course and Unit of Study Portal (CUSP) work was recently presented at the Seventh International
The Course and Unit of Study Portal (CUSP) is a Computing Education Research Workshop (Sheard et al
software product that was developed jointly by three 2011) and another is being presented at ACE 2012
faculties of the University of Sydney as a university- (Simon et al 2012).
funded project to provide a common curriculum mapping Properties encoded about an exam question in the
framework for a diverse range of professional degrees current draft of the classification include type of question
across Engineering, IT, Architecture, Design, Urban (e.g. short answer, multiple choice), topics examined (e.g.
Planning, and Health Sciences. CUSP is currently used at data types, loops, OO concepts, program design), type of
the University of Sydney for over 240 degrees and over skill required (e.g. knowledge recall, hand executing
2,500 units of study across four faculties. (Note: the code, writing code, explaining code), and difficulty (high,
University of Sydney uses the term ‘unit of study’ for medium, low).
what some other universities call a ‘subject’ or a ‘course’, Getting academics to agree on classifications of
and the term ‘course’ for what some other universities specific questions has not proved to be straightforward.
call a ‘degree’ or a ‘program’.) For example, two of the project participants recently
CUSP captures the representation of multiple sets of classified an introductory programming exam consisting
graduate attributes and accreditation competencies entirely of multiple-choice questions. While computing
(named curriculum goals or curriculum goal frameworks) academics are divided on the value and validity of
and maps these to the relevant degrees. Each degree multiple-choice questions (Shuhidan et al, 2010), they are
structure is modelled into the system as a collection of nevertheless widely used (Simon et al, 2012). On the
core subjects plus the rules governing the selection of issue of degree of difficulty (high, medium, low) the two
elective subjects. Each graduate attribute or accreditation participants agreed independently on only one third of the
competency is in turn mapped to each assessment and multiple-choice questions. On skill required (e.g.
learning outcome within each subject of a degree. This knowledge recall, hand executing code, explaining code)
design enables the CUSP system to generate reports that they agreed independently on one quarter of the multiple-
visualize the curriculum coverage for entire degrees choice questions. It is hardly surprising that subject
against any of the curriculum goal frameworks attached. outlines and other documents are ambiguous, when two
These reports in turn enable quick identification of any experienced teachers of introductory programming
gaps in goal coverage or any sequencing problems in the exhibit such a low level of agreement on a set of
degree structure and facilitate accreditation or other multiple-choice questions. Before there can be a
quality control review processes. This is described in substantive debate on the content and assessment of early
greater detail by Gluga et al (2010). programming courses, there needs to be greater consensus
Richard Gluga, a PhD candidate at the University of on a framework for the debate – a framework that this
Sydney, is creating an enhancement of CUSP, known as project aims to provide.
ProGoSs – Program Goal Progression (Gluga et al, 2012),
which can be used to map the detailed objectives for the 2.4 Neo-Piagetian Theory
programming fundamentals curriculum designed by the Wright, Hadgraft and Cameron (2010) describe a
ACM/IEEE (2008). The extension is intended to support dialectic in learning outcomes, with one part of the
systematic design, modelling and monitoring of student dialectic being a “list of discrete outcomes or
progression as part of curriculum design using Bloom's aspirational statements” as opposed to the other part of
Taxonomy (Bloom, 1956) and neo-Piagetian cognitive the dialectic, “threshold learning outcomes [that] reflect
development theory (Lister, 2011). It also supports the way engineers and ICT professional approach, think
curriculum design by allowing for the specification of the and do their work”. In this project we adopt a cognitive
level of achievement of both higher- and lower-achieving development perspective to transcend that dialectic.
students, so that institutions can design a curriculum, and Piaget developed a very well known constructivist
assess how well it is achieving its learning outcomes, theory about the different levels of abstract reasoning
with full regard to the range of achievement of the exhibited by people as they mature from child to adult.
students who complete degrees. While classical Piagetian theory has been largely
abandoned, neo-Piagetian theory has overcome many of
2.3 Exam Question Classification the problems that led to that abandonment. The types of
The aim of the Exam Question Classification project is to abstract reasoning are broadly the same in both theories;
investigate the nature and composition of formal but in neo-Piagetian theory, people, regardless of their
examination instruments used in summative assessment age, are thought to progress through increasingly abstract
of introductory programming students, and the forms of reasoning as they gain expertise in a specific
pedagogical intentions of the educators who construct problem domain. Neo-Piagetians attribute the increasing
these instruments. The project leaders presented their first abstraction in reasoning not to biological maturity but to
draft of a classification scheme in a half-day workshop at an increase in the effective capacity of working memory,
the 2011 ACE Conference in Perth. On the basis of the as the learner ‘chunks’ knowledge. Neo-Piagetian theory
feedback received from the 20 or so workshop is not esoteric – the popular SOLO taxonomy (Biggs and
participants, the project leaders revised their initial Collis, 1982) is based upon neo-Piagetian theory.
scheme. Subsequently, project members formed pairs and In a paper presented at the 2011 Australasian
applied the revised scheme to analysing a total of twelve Computing Education Conference, Lister (2011)
exams, from nine different universities in Australia, the proposed a way of applying neo-Piagetian theory to the
UK, New Zealand, Finland and the USA. A paper on this learning of programming. He defined the development of
the novice programmer in terms of three neo-Piagetian proposals is not accepted, alternative dissemination
stages. At a pre-operational stage, students can trace the mechanisms will be formulated.
changing values in a piece of code, but do not reason in The workshops will be open to all interested
terms of abstraction of that code. At a concrete academics, and will probably not require a registration
operational stage, students can reason in terms of fee.
abstractions, but only in the context of specific code. At a The budget allocation for dissemination and evaluation
formal operational stage, students can reason in terms of workshops includes a limited number of ‘scholarships’
programming abstractions without recourse to explicit that will pay the registration fee for ACE 2013, to be held
code examples. Lister’s stage theory has already been in Adelaide. These scholarships will be awarded to
adopted by CUSP participants at the University of people outside the project who contribute documents,
Sydney, and empirical results from Queensland data or other material that manifestly advances the
University of Technology (Corney et al, 2012; Teague et project.
al, 2012) add support to the proposal. The ITiCSE working group reports are among the
most influential and highly cited papers in computing
3 Dissemination Strategy education. Thus an ITiCSE working group in 2013 will
There is very little point to this project, or to any other maximise the potential for international dissemination.
innovative, education-related project, if the outcomes of
the project remain private to the direct project 3.1 Monthly Meetings
participants. In many respects, the success of any As part of the project, full-day meetings will be held each
innovative, education-related project should be assessed month in at least two of Melbourne, Sydney and
by the degree of dissemination of the outcomes. Brisbane.
By ‘dissemination’, we do not simply mean the The project values collaboration, so these meetings are
distribution of information via publications and seminars not necessarily closed, and researchers not currently
(although distribution of information is an essential involved in the project may be invited to attend them.
component of a successful dissemination). For the However, while the six-monthly workshops are open to
authors of this paper, ‘dissemination’ is to be measured anyone even if they merely wish to observe, an invitation
by the extent of adoption by others of the materials and to a monthly meeting will be made on the assumption that
techniques developed by the authors of this paper. the invitee will play an active and continuing role. For the
It is well documented that dissemination, as the term is types of active roles that are suitable, see Section 5.1,
used in this project, is difficult. Few innovative, ‘Rules of Engagement’. A person seeking to join the
education-related projects have succeeded at project under this arrangement may need to make an
dissemination (Gannaway et al, 2011; McKenzie et al, explicit time commitment. A 10% time commitment is
2005; Southwell et al, 2005 & 2010). To improve roughly two days a month, and with such a level of
dissemination, the ALTC explicitly adopted a commitment a person might spend one of those days at a
Dissemination Framework (ALTC, 2006), which has also project meeting in their own city and the other day
guided the authors of this paper. Even with the untimely working independently to prepare for the next meeting.
demise of the ALTC, this dissemination framework is
likely to influence the design of Australasian education 4 Project Organisation
projects well into the future. This project unifies three existing projects, spread across
As advocated within the ALTC Dissemination six universities in three Australian states:
Framework, this project has adopted an ‘engaged’ model
• Exam Question Classification: As described earlier,
for dissemination:
this sub-project is investigating the nature and
“involving consultation, collaboration and composition of formal examination instruments used
support for ongoing dissemination both during the in summative assessment of introductory
project and after the project is completed” programming students, and the pedagogical intentions
of the educators who construct these instruments.
Consequently, the dissemination of this project begins
early in the project (indeed, it begins with the publication • Syllabus Specification: This sub-project builds upon
of this paper) and will continue throughout the project, the CUSP system discussed earlier, to create a new
based on proposed full-day workshops held at roughly system into which we can map the detailed curriculum
six-monthly intervals in conjunction with major of programming subjects.
computing education conferences: • Exam Question Generation and Benchmarking:
• ACE 2012 (January, Melbourne) This sub-project aims to include some common
questions in exams at participating institutions, and to
• ICER 2012 (August, Auckland) benchmark student performance on those questions.
• ACE 2013 (January, Adelaide)
Each month there will be up to three one-day meetings in
• ITiCSE 2013 (June/July, Canterbury, UK). Brisbane, Sydney and/or Melbourne. The ‘major’
While the first of these workshops is now confirmed, the meeting will be attended by all participants from the city
other three will be subject to proposal and acceptance at where it is held, along with one representative from each
the respective conferences. As the workshops also serve participating institution outside that city. One or two
to define project stages and milestones, if any of the further meetings will be smaller, involving just the
project leader and the project participants located in that to warrant recognition as an author? In what order should
city. the authors’ names be listed? To these and similar
questions Lister and Edwards (2010) propose answers
4.1 Rules of Engagement that might well be adopted for the BABELnot project.
All project participants have agreed to the following rules
about how they will work together: 5 Evaluation Framework
• All members will help to procure the outlines and A requirement of ALTC-funded projects of this size is
other public documents for the relevant programming that they have an external reference group and be
subjects at their respective universities. formally evaluated by an outside evaluator. This is not an
activity that we plan to leave until the end of the project.
• All members will assist in rewriting those documents Instead, formative evaluation throughout will facilitate
into the CUSP-derived system adopted by the project. the attainment of better project outcomes.
• Project members who teach one of the programming While the most obvious role of the six-monthly
subjects will: workshops is dissemination, evaluation will also figure
o Provide other project members with the prominently at all the workshops. For example, by having
opportunity to run short, in-class, formative workshop attendees perform training exercises such as
assessment exercises. classifying exam questions in the CUSP-derived
formalism, we will collect evaluation data on whether the
o Give full consideration to using in their summative
formalism can be understood easily and applied reliably.
tests and exams the questions devised as part of
The external evaluator will be appointed six months
this project. However, the final decision on the
after the project begins, and will then work with the
inclusion of any summative test or exam question
project team to develop an evaluation plan. The key
remains with that teacher.
sources of information for the evaluation will be: reports
o Consider contributing some of their own questions from the monthly meetings, dissemination events, and
to a common, public pool of questions. reference group review meetings; interviews with project
• All members will approach the (other) teachers of the members; and feedback sought from project members via
first three programming subjects at their institutions, surveys.
to request test papers, exam papers, and class A working group will be proposed for the ITiCSE
performance data on those papers. Ideally, the papers conference to be held in the UK in mid-2013. If the
thus solicited will be public domain, but the project proposal is accepted, this working group will contribute
will also accept and use papers on the understanding to the summative evaluation of the project. An ITiCSE
that they are to remain confidential. working group will provide a fresh set of academics,
• At least one member at each participating institution independent of the formative evaluations.
will complete any necessary ethical clearance process.
6 Conclusion
• Members interested in particular sub-projects will This paper represents a break in tradition, driven by our
undertake tasks related to those sub-projects. focus on dissemination. Traditionally, innovative
Specifically: education projects focus on their product, and do not
o Members most interested in syllabus specification report on their activities until either those activities are
will benchmark the enhanced CUSP system by over, or at least a significant milestone has been attained.
encoding information about subjects at other We describe that as the ‘disseminate late’ approach, and
institutions. argue that this traditional approach has probably
o Members most interested in the exam question contributed to poor dissemination outcomes. Instead, we
classification/archive will develop meta-tags for advocate a ‘disseminate early and often’ approach, which
the classification/archive that has CUSP as its is what we are implementing in this project, primarily
starting point, and will archive questions and through the six-monthly workshops, but also through the
information from the benchmarking of questions. very act of writing this paper − we are like software
They will also conduct and analyse interviews engineers who advocate writing the software tests early,
with relevant academics to explore the processes even before writing the code, for well known reasons that
of writing and marking programming exams. we believe are analogous to why innovative education
projects should begin dissemination early.
o Members most interested in question Through beginning the dissemination early, we now
benchmarking will provide their questions and have the luxury of using this paper to invite others to
student performance data for the classification/ participate in this project. Most may elect to attend any of
archive. the six monthly workshops, but others may accept our
• In addition to the specific tasks listed above, all invitation to take on a more active role, and join us in the
members will make some contribution across all parts monthly meetings.
of the project, and will actively pursue connections It is just over 10 years since McCracken et al’s (2001)
between the sub-project of primary interest to them paper appeared. That paper directly and indirectly
and the other sub-projects. inspired a raft of multi-institutional collaborations in
Further matters still to be agreed by the participants computing education. However, most of those projects
include a protocol for authorship in papers produced were either short-lived, or (like BRACElet) were loosely
within the project. What level of contribution is required organised, with much of the work being done without
formal financial support. Our project may be a prototype Clear, T, J Whalley, P Robbins, A Philpott, A Eckerdal,
for new generation of multi-institutional projects. Our M-J Laakso, & R Lister (2011). Report on the final
project is formally funded, and with that comes a BRACElet workshop: Auckland University of
commitment to deliver. That in turn leads to a more Technology, September 2010. Journal of Applied
formally organised project structure. Also, applying Computing and Information Technology 15(1).
lessons that have been learnt over the last ten years, the Corney, M, D Teague, A Ahadi, & R Lister (2012). Some
project has dissemination built explicitly into its structure. empirical results for neo-Piagetian reasoning in novice
As previously mentioned, this project builds upon the programmers and the relationship to code explanation
University of Sydney’s existing CUSP system, which is questions. 14th Australasian Computing Education
currently used for over 240 degrees across four faculties Conference (ACE 2012), Melbourne, Australia.
at that institution. It is therefore possible that, after this
Gannaway, D, T Hinton, B Berry, & K Moore (2011). A
project is formally over, the extended version of CUSP
review of the dissemination strategies used by projects
would be adopted by the current users: a cycle of
funded by the ALTC Grants Scheme. Sydney:
innovation would be completed, by applying the
Australian Learning and Teaching Council.
outcomes and deliverables of this project to disciplines
other than computing. Gluga, R, J Kay, & T Lever (2010). Modeling long term
learning of generic skills. Tenth International
7 The Name BABELnot Conference on Intelligent Tutoring Systems,
Here is a brief explanation for readers who are curious Pittsburgh, PA, USA, 85-94.
about the name BABELnot. The name is deliberately Gluga, R, J Kay, R Lister, S Kleitman, & T Lever (2012).
reminiscent of the BRACElet project, of which Raymond Coming to terms with Bloom: an online tutorial for
Lister was a leader. The capitalisation of BRACElet teachers of programming fundamentals. 14th
reflects its continuation from the BRACE project, in Australasian Computing Education Conference (ACE
which the name BRACE was an acronym. The name 2012), Melbourne, Australia.
BABELnot calls to mind the biblical tale of the Tower of Lister, R (2011) Concrete and other neo-Piagetian forms
Babel, whose builders failed in their task because they of reasoning in the novice programmer. 13th
lost the ability to communicate with one another. So Australasian Computing Education Conference, Perth,
while Babel highlights the challenge of communication Australia, 9-18.
when we speak different languages, BABELnot will help
overcome that challenge by devising a common language Lister R, ES Adams, S Fitzgerald, W Fone, J Hamer, M
in which programming educators may better Lindholm, R McCartney, JE Moström, K Sanders, O
communicate with one another on matters of standards Seppälä, B Simon, & L Thomas (2004). A multi-
and assessment within their subjects. national study of reading and tracing skills in novice
programmers. SIGCSE Bulletin 36(4):119-150.
Acknowledgements Lister, R, T Clear, Simon, DJ Bouvier, P Carter, A
This project was approved for funding by the Australian Eckerdal, J Jacková, M Lopez, R McCartney, P
Learning and Teaching Council, and is funded under the Robbins, O Seppälä, & E Thompson (2010). Naturally
auspices of the Australian Federal Government’s occurring data as research instrument: analyzing
Department of Education, Employment and Workplace examination responses to study the novice
Relations (DEEWR). However, the views expressed in programmer. SIGCSE Bulletin 41(4):156-173.
this paper are solely those of the authors, and not the Lister, R & J Edwards (2010). Teaching novice computer
views of the ALTC or DEEWR. programmers: bringing the scholarly approach to
Australia − a report on the BRACElet project.
References Australian Learning and Teaching Council.
ACM/IEEE (2008). Computer Science Curriculum 2008: Lister, R, C Fidge, & D Teague (2009). Further evidence
An Interim Revision of CS 2001. http://www.acm.org/ of a relationship between explaining, tracing and
education/curricula/ComputerScience2008.pdf . writing skills in introductory programming. 14th
Australian Learning and Teaching Council (2006). ALTC Conference on Innovation and Technology in
Dissemination Framework. http://www.altc. edu. Computer Science Education (ITiCSE 2009), Paris,
au/resource-dissemination-framework-altc-2008 France, 161-165.
Biggs, JB & KF Collis (1982). Evaluating the quality of McCracken, M, V Almstrum, D Diaz, M Guzdial, D
learning: The SOLO taxonomy (Structure of the Hagen, Y Kolikant, C Laxer, L Thomas, I Utting, & T
Observed Learning Outcome). New York: Academic Wilusz (2001). A multi-national, multi-institutional
Press. study of assessment of programming skills of first-year
Bloom, BS (1956). Taxonomy of Educational Objectives: CS students. SIGCSE Bulletin 33(4):125-140.
Handbook I: Cognitive Domain. Longmans, Green and McGettrick, A, R Boyle, R Ibbett, J Lloyd, L Lovegrove,
Company. & K Mander (2005). Grand challenges in computing:
Carter, J, J English, K Ala-Mutka, M Dick, W Fone, U education – a summary. The Computer Journal
Fuller, & J Sheard (2003). How shall we assess this? 48(1):42-48.
SIGCSE Bulletin 35(4):107-121. McKenzie, J, S Alexander, C Harper, & S Anderson
(2005). Dissemination, adoption and adaptation of
 project innovations in higher education. Sydney: Computing Education Conference (ACE 2012),
University of Technology, Sydney. Melbourne, Australia.
Sheard, J, Simon, M Hamilton, & J Lönnberg (2009). Southwell, D, D Gannaway, J Orrell, D Chalmers, & C
Analysis of research into the teaching and learning of Abraham (2005). Strategies for effective dissemination
programming. Fifth International Workshop on of project outcomes. Carrick Institute for Learning and
Computing Education (ICER 2009), Berkeley, CA, Teaching in Higher Education.
USA, 93-104. Southwell, D, D Gannaway, J Orrell, D Chalmers, & C
Sheard, J, Simon, A Carbone, D Chinn, M-J Laakso, T Abraham (2010). Strategies for effective dissemination
Clear, M de Raadt, D D’Souza, J Harland, R Lister, A of the outcomes of teaching and learning projects.
Philpott, & G Warburton (2011). Exploring Journal of Higher Education Policy and Management
programming assessment instruments: a classification 32(1):55-67.
scheme for examination questions. Seventh
Teague, D, M Corney, A Ahadi, & R Lister (2012).
International Computing Education Research
Swapping as the ‘Hello World’ of Relational
Workshop (ICER 2011), Providence, RI, USA, 33-38.
Reasoning: Replications, Reflections and Extensions.
Shuhidan, S, M Hamilton, & D D’Souza (2010).
14th Australasian Computing Education Conference
Instructor perspectives of multiple-choice questions in
(ACE 2012), Melbourne, Australia.
summative assessment for novice programmers.
Computer Science Education 20:229-259. Whalley, J, R Lister, E Thompson, T Clear, P Robbins,
PKA Kumar, & C Prasad (2006). An Australasian
Simon, A Carbone, M de Raadt, R Lister, M Hamilton, &
Study of Reading and Comprehension Skills in Novice
J Sheard (2008). Classifying computing education
Programmers, using the Bloom and SOLO
papers: process and results. Fourth International
Taxonomies. Eighth Australasian Computing
Workshop on Computing Education (ICER 2008),
Education Conference (ACE 2006), Hobart, Australia,
Sydney, NSW, Australia, 161-171.
243-252.
Simon, J Sheard, A Carbone, D Chinn, M-J Laakso, T
Wright, S, R Hadgraft, & I Cameron (2010). Learning
Clear, M de Raadt, D D’Souza, R Lister, A Philpott, J
and Teaching Academic Standards Project:
Skene, & G Warburton (2012). Introductory
Engineering and ICT. Australian Learning and
programming: examining the exams. 14th Australasian
Teaching Council.