See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/259440606

Success Failure or No Significant Difference

Article  in  International Journal of Emerging Technologies in Learning (iJET) · March 2013

DOI: 10.3991/ijet.v8i1.2376

CITATIONS READS

4 554

1 author:

Mary Burns
Education Development Center
56 PUBLICATIONS   435 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

INEE Teacher Professional Development in Crisis View project

Preparing All Learners View project

All content following this page was uploaded by Mary Burns on 24 December 2013.

The user has requested enhancement of the downloaded file.

 PAPER
SUCCESS, FAILURE OR NO SIGNIFICANT DIFFERENCE: CHARTING A COURSE FOR SUCCESSFUL EDUCATIONAL TECHNOLOGY INTEGRATION

Success, Failure or no Significant Difference:

Charting a Course for Successful Educational
Technology Integration
http://dx.doi.org/10.3991/ijet.v8i1.2376

Mary Burns
Education Development Center, Waltham, MA, USA

Abstract—The question of whether computers have posi- The second camp, which I term techno-enthusiasts, sees
tively or negatively impacted student learning is still hotly technology as essential to increasing modernization and
contested in educational technology circles, particularly in efficiency in schools. They may regard technology as
the area of international development, by proponents and tools of educational reform, in particular compensating for
critics of technology in education. Overall, research offers (or even eliminating) poor teachers and teaching. They
conflicting answers to this question. However, the abundant may view technology as a catalyst for propelling students
research on effective school change and innovation imple- toward 21st century learning [3]. Many techno-enthusiasts
mentation points to practices which those who promote see computers as tools for student engagement, indirectly
technology in schools should tap. This paper outlines the resulting in learning gains and directly resulting in such
long-term structural conditions that can lead to the deep qualities as increased confidence and self-efficacy. A
change technology initiatives seek to promote. small but significant group consists of educational tech-
nology utopians who see computers revolutionizing the
Index Terms—Benefits and weaknesses of ICT in education, educational paradigm. Though many acknowledge the
educational technology debate, ICT in education dearth of consistent, large-scale research showing a causal
relationship between technology and student learning,
I. INTRODUCTION they indict the poverty or poor quality of human and
structural supports versus limitations with the technology
Have computers positively or negatively impacted stu- itself.
dent learning? Responses to such a broad and binary In fairness to both “camps,” each argument is more
question often serve as sort of a Rorschach test of respon- nuanced than reported here. Further, debates around
dents’ beliefs about the potential and actual benefits of classroom computers are often more multipolar than
technology as a reform tool; the responsibilities of schools bipolar. Nonetheless, arguments can be distilled to the
and teachers vis-à-vis technology use and adaptation; and same competing sentiments toward computers as student
organizational and human change processes. learning tools. Though the temptation is to ask, Which
The rhetoric around educational technology, particu- camp is right? it may be more useful to examine the
larly in non-industrialized nations, but in industrialized substance of issues raised by each side. Techno-
country contexts as well, is energetic and robust—as the enthusiasts are correct in stating that the issues that plague
most cursory reading of blogs and discussion boards on “schooling” are human and organizational and not derived
the subject will attest. Broadly and dichotomously, the from technology per se. On the other hand, as techno-
two camps may be defined as techno-dissenters who dissenters rightly note, there are abundant examples of
believe the impact of technology on education, particu- failed technology initiatives and school computer use that
larly on teaching and learning, has been “oversold” [1] diverts from, rather than enhances, student learning.
and unproven. Such techno-dissenters may be former
technology enthusiasts who have been chastened by a lack II. COMMON GROUND
of unequivocal longitudinal research demonstrating
causality between technology use and student achieve- Yet both sides are closer to agreement than appears at
ment and who are thus skeptical of reputed benefits. first blush. Both camps agree that the issues that beset
Among the ranks of such detractors are those who rail computers as learning tools have little to do with technol-
against what they see as the threat of “technopoly” [2]— ogy per se—tools themselves are but artifacts of human
an overemphasis on rationalization, quantification and attitudes, values and needs. Both techno-dissenters and
efficiency that belies arguments for educational technol- techno-enthusiasts, through their attack on computers’
ogy use and overly technocratic prescriptions for what are failure to cure what ails schools and their indictment of
often human and structural maladies in schools. The school environments that thwart the vast potential of
suggested remedies of techno-dissenters range broadly technology, respectively, essentially point to the same
from proposing a moratorium on or cuts in technology issue—the failure of educational technology initiatives are
spending in schools, to greater reflection and examination structural and human in nature and thus any curative
about technology use in schools, to a cooling of the measures must address these human and organizational
rhetoric around technology’s learning potential. issues.
Accordingly, a preponderance of research evidence
suggests that under certain conditions, technology is

38 http://www.i-jet.org
 PAPER
SUCCESS, FAILURE OR NO SIGNIFICANT DIFFERENCE: CHARTING A COURSE FOR SUCCESSFUL EDUCATIONAL TECHNOLOGY INTEGRATION

correlated with improved student academic performance six levels within the cognitive domain, from simple recall
across a range of content areas [4]. This is where the or recognition of facts and comprehension of these facts at
arguments of techno-dissenters and techno-enthusiasts can the lowest level, through increasingly more complex and
fuse. Rather than focusing on technology solutions, we abstract mental levels—or “higher-order thinking”— such
must turn these conditions into the “right” conditions. as application, analysis, synthesis, and evaluation [6].
This article uses the competing camps of techno- These three conditions for knowledge construction (and
dissenters and techno-enthusiasts as a launch pad to these are by no means the only three)—learning as a
explore conflicting research around technology’s impact product of interaction with rich stimuli; learning as a
on student learning. It begins by summarizing the essential continual, developmental, evolutionary process; and the
link between new notions of learning and teaching and dialectic of learning (knowledge and concepts constructed
their nexus with technology (I use “technology” and through the interplay of various factors) have spawned a
“computers” synonymously in this article). I argue that new focus on instruction and on pedagogical approaches.
where we’ve seen teachers use technology to help students The traditional transmission model (e.g., lecture/short
learn in ways that not only support, but make possible answer format), with its emphasis on quantity, coverage
learning that would otherwise be impossible, it is because and product creation has yielded, or is at least losing
nations or districts or schools have cultivated the far more ground, to learner-centered approaches with their focus on
difficult human and structural infrastructure necessary for learning as a process that must be examined and under-
this success. I conclude by proposing a number of requi- stood.
site scaffolds or conditions, primarily for non- Thus, in many classrooms across the globe we have
industrialized country contexts, necessary for the conver- seen increased attention on new conceptual structures and
gence of technology and learning to truly occur. understanding of complex and often conflicting informa-
tion. Mindful of the new body of knowledge on learning,
III. TEACHING, LEARNING AND TECHNOLOGY: AN we notice greater efforts to purposefully utilize learner-
OVERVIEW centered pedagogy—an instructional approach in which
We have made great strides in the past several decades students explore, manipulate, question, and discover
in understanding how learning occurs. We know that answers for themselves. We observe attempts to create
learning is not an isolated or static process. As we interact activities that are developmentally appropriate, yet chal-
with the world around us, and the infinite variety of lenging enough to allow for a certain level of frustration
images, ideas, information, and other stimuli that com- on the part of the learner. In short, to use Bloom’s taxon-
prise our world, we are constantly constructing, revising, omy once again, we see attempted convergence between
and reconstructing our knowledge and beliefs to create a instruction, curriculum, and tools in the promotion of
new framework of understanding. Knowledge then is higher order learning.
constantly under construction—a dynamic, evolutionary,
developmental process. IV. TECHNOLOGY AND STUDENT LEARNING
We know too that learning is influenced by our level of Since learning occurs by interaction with rich stimuli, is
biological and psychological development. As the writings a developmental process, and generates new knowledge
of Jean Piaget [5] assert, children think and reason differ- that challenges, adds to, or deepens the learner’s existing
ently at various periods in their lives, passing through a framework of knowledge, computer technology would
series of stages in their cognitive formation—from the appear to be a good fit with the above paradigm. When
sensorimotor stage, during which the child gains motor used appropriately, technology can become a “mind tool,”
control, through to the formal operational stage, where functioning, in Jonassen’s words [7] as an intellectual
the child begins to reason logically and systematically. partner with the learner to engage and facilitate critical
And we know that learning is oftentimes fraught with thinking and higher-order thinking.
tension and conflict. If new information matches our We have embraced technology’s potential to help stu-
existing understanding, we can easily assimilate it. How- dents scale the levels of intellectual development. Indus-
ever, if new information does not match our existing trialized nations have invested millions of dollars to make
framework—or threatens our existing corpus of knowl- such “mind tools” accessible to most, if not all, learners.
edge—we must accommodate it, either by forming new Donors, foundations and government agencies in non-
understandings, or re-evaluating our prior beliefs and industrialized countries have provided computer labs,
reconstructing our prior theories, or reject that new infor- teacher training and in some cases, “one laptop per child”
mation. This continuous struggle between pieces of to extend the purported learning benefits of technology to
varying and oftentimes conflicting information—this the world’s poorest teachers and students. This move to
dialectic of learning— occurs constantly, sometimes provide access to all teachers and learners rests on the
consciously, or more often, unconsciously, and contributes belief and hope that computer technology, by its very
to our overall construction of knowledge. Without this multichannel and interactive nature, is an ipso facto
disequilibrium, the student’s belief system is not chal- learning tool, and that by employing computers in a
lenged and the potential for greater intellectual growth is manner consistent with what we know about best teaching
stifled. Learning then is rarely a final product. More often and learning practices, students will be more likely to
it is a constant, evolutionary, and sometimes revolution- attain such higher order thinking, 21st century learning or
ary, process. simply just demonstrate learning gains.
The idea of learning as a developmental process has
also been firmly established in a number of educational
systems across the globe through learning “taxonomies.”
The most well-known—Bloom’s Taxonomy—identified

iJET – Volume 8, Issue 1, March 2013 39

 PAPER
SUCCESS, FAILURE OR NO SIGNIFICANT DIFFERENCE: CHARTING A COURSE FOR SUCCESSFUL EDUCATIONAL TECHNOLOGY INTEGRATION

V. COMPUTERS AND LEARNING: THE GOOD, THE BAD, THE open-ended writing prompts that require students to
CONTRADICTORY AND THE INCONCLUSIVE generate responses using paper and pencil underestimate
the achievement of 4th and 8th grade students used to
Yet the outcomes of this long hoped-for convergence of writing with computers [15]. Consistent with these find-
technology and learning are confounding. Hypotheses ings is a body of research since the 1980s demonstrating a
supporting or refuting the link between technology and consistent link between word processing and improved
learning remain inconclusive and contradictory as the writing scores [16].
following selection of well-known and large-scale re-
search on technology and learning attest.  Links between “e-maturity” and student perform-
ance scores: The British Educational Communications
A. Positive Benefits and Technology Agency (BECTA) reported that schools
Research increasingly notes the relationship between with good ICT resources, such as high-speed broadband
student computer use and increased academic perform- access and interactive whiteboards, achieve better results
ance (primarily gauged through national or international in national tests taken at age 16. Interactive whiteboards,
examinations). A quick encapsulation of some cross-study in particular, appear to result in improved test perform-
findings are noted here: ance for low-achieving students particularly in writing,
math and science [17].
 Benefits of technology on increased math and
science content knowledge: One of the most well-known B. Negative Results
meta-analyses on the connection between computers and There are also a number of well-known studies that
student learning is that of Kulik [8] who noted that stu- demonstrate a negative link between student computer use
dents who used computer tutorials in mathematics, natural and learning.
science, or social science scored significantly higher in
these subjects compared to traditional approaches—  Higher scores for students in face-to-face versus
equivalent to an increase from 50th to 72nd percentile in online learning situations: One study [18] compared
test scores. Similar findings suggesting the link between student scores in face-to-face and online economics
technology use and improved math and science scores can courses taught at three different institutions. After taking
be found in international studies [9] [10]. Kulik’s meta- into account selection bias and differences in student
analysis also revealed that students who used simulation characteristics, they reported that the average scores were
software in science also scored higher, equivalent to a almost 15 points higher for the face-to-face format than
jump from 50th to 66th percentile. Consistent with Kulik’s for the online format.
meta-analysis, Integrated Learning Systems (ILS), which  Higher scores for non-users of technology: An
usually rely heavily on tutorial instruction, have been Israel study [19] analyzed the effects of a large-scale
producing positive results in mathematics programs for computerization policy in elementary and middle schools.
decades. Computer tutorials in natural and social science Treatment schools received technology. Control schools
classes also have had an almost uniformly positive record did not. The researchers found evidence that increased
of effectiveness in the1970s, 1980s, and 1990s [11]. educational use of computers did not raise test scores.
 Benefits of technology on students’ science scores Rather, they found a negative and significant relationship
in PISA: The Organization for Economic Co-operation between the program-induced use of computers and the
and Development (OECD) has for the past several years 4th grade maths scores.
tracked the link between student computer use and student  Higher levels of student achievement for students
scores on its Programme for International Student As- lacking access to computers: Fuchs & Wößmann’s oft-
sessment (PISA). Analysis of 2006 student science scores cited study shows a positive correlation between student
by the OECD suggests that student performance in science academic achievement and computer availability both at
is positively related to length of computer use. Students home and at school. However, once researchers controlled
from OECD countries who have used computers for more extensively for family background and school characteris-
than five years, and are therefore more familiar with tics, the relationship became negative for home computers
computers, score at the middle and higher end of Level 3 and insignificant for school computers. The authors
in science in PISA versus students who have been using concluded that “mere availability of computers at home
computers for less than two years, who score at the middle seems to distract students from effective learning” [20].
or low end of level 2. The difference between these two
scores is 90 points or more one entire level of proficiency C. Contradictory results
[12]. However, the report notes that greater computer use Increasingly, as evaluations become more rigorous,
is tied to higher socioeconomic status which in turn is there appears to be a good deal of contradictory evidence
linked to higher academic achievement. about technology’s impact on student learning. Much of
 Improved writing: US students who used word this contradiction is grounded in the nature and size of the
processors or otherwise used computers for writing scored studies. Further, the divergence in results often breaks
higher on measures of writing skill, equivalent to a rise along lines of socioeconomic, gender, student achieve-
from the 50th to 62nd percentile [13]. Also in the United ment, and ethnic differences.
States, Russell & Plati [14] examined 4th and 8th grade  Different achievement results with different
students who had access to digital writing tools such as a populations: A 2010 OECD study finds that computers
laptop, an E-Mate or an Alpha Smart. These students— “amplify” the differential between richer and poorer
even when they took the test using paper and pencil—did science students in Finland, Ireland, Spain and Switzer-
better on state writing tests than their peers who had land while also ameliorating this gap among rich and poor
access to no such digital tools. The study’s authors con- science students in Canada, the Czech Republic, New
cluded that the mode of test administration is important— Zealand, Poland and the Slovak Republic [21].

40 http://www.i-jet.org
 PAPER
SUCCESS, FAILURE OR NO SIGNIFICANT DIFFERENCE: CHARTING A COURSE FOR SUCCESSFUL EDUCATIONAL TECHNOLOGY INTEGRATION

 Lack of consensus around technology’s benefits in VI. TECHNOLOGY IN SCHOOLS: THE “RIGHT” CONDITIONS
science achievement: The OECD’s 2010 report on The concerns of techno-dissenters are not simply aca-
student computer use and its relationship to PISA scores demic or philosophical but financial and educational.
contains several contrasting statements. In one part of the Money is a finite resource. Governments and donors have
report, authors note that computers are “not necessarily expended enormous amounts of money for technology in
more beneficial for students in subject-based assessments” the hope that computers can reform schools in ways that
[22] and “Higher performance in science is related to other initiatives have been unable to and many nations are
lower educational use of computers” [23]. These state- either poised or considering huge investments in educa-
ments are in direct contrast to other parts of the report tional technologies as part of economic planning. There is
which link computers to positive attitudes and perform- evidence [32] that such spending places greater financial
ance in science [24]. burdens on poorer and middle income countries than on
 Conflicting results about the same types of tech- rich ones. Similarly, the education sector has a long
nology interventions: Some studies of online instruction history of investing in innovations that are often discred-
and its impact on student learning show no consensus on ited. And, as techno-dissenters imply, a great deal of
the impact of the same technology intervention. For fetishism characterizes any use of technology in schools.
instance, in the US, Cavanaugh found “equivalence” But again the two sides’ arguments can be reconciled on a
between face-to-face and online learning [25]; while practical level. As both camps would agree, donors,
Shachar & Neumann found “significant positive effects” ministries and schools have more often than not neglected
for online learning over face-to-face instruction [26]. In to establish the conceptual, organizational, and instruc-
contrast, Ungerleider & Burns noted “no significant tional and evaluator framework—the “right” conditions—
difference” between the two [27]. in which teachers and students can succeed so that the
rich, complex and intricate kinds of learning with technol-
D. No difference or inconclusive evidence ogy, described earlier can begin to form and flourish.
Particularly with newer technologies and newer innova-
tions, such as virtual schools or tablets, there is often a A. Policy and Programmatic Changes
lack of what the US Department of Education’s What Teachers, not technology, are essential to student learn-
Works Clearinghouse calls “acceptable” standards of ing—and teachers do not operate within a vacuum.
evidence. Rather, they often function within multi-layered, matryo-
 No significant difference: Another OECD report shka-like educational systems. This larger system includes
shows that computer use in school shows no significant policies and the beliefs such policies implicitly espouse
difference in students’ scores on the PISA. Austria is the about how children and adults best learn and work;
only country where computer use at school had a larger associated practices to disseminate such policies; the
effect on science scores than at home (3 points) [28]. establishment of curricular, assessment and evaluation
 Correlation but not causation: Britain’s ImpaCT2 systems to implement such practices; attitudes about the
study of over 2000 Key Stage 2 and 3 pupils from 1999- professionalism and capacity of teachers and principals to
2002 showed that ICT had a positive relationship to implement such practices; and beliefs about how technol-
students’ learning of mathematical skills linked to the ogy should be used, for what purposes and by whom. This
amount and type of use of ICT in the mathematics curricu- is the superstructure within which educational technology
lum. High users of ICT at KS3 outperformed, on average, initiatives must operate, with which they must conform,
low users of ICT in mathematics, but differences at KS4 and against which they must contend. As the title of Larry
were slight. However, this aspect of the research looked Cuban’s book, Computers Meet Classroom; Classroom
only at correlation and not causation [29]. Wins adumbrates, the organizational status quo can often
neuter even the most dynamic and hopeful of innovations.
 Conflation of computers and socioeconomic status:
Computer use is often used as a proxy for individual If, as techno-enthusiasts, hope, and techno-dissenters
household wealth. Researchers [30] [31] point to wealth as lament, computers have any hope of becoming tools of
a greater determinant of student academic performance learning, the larger educational landscape must be culti-
than access to and use of computers. Where computers vated or more accurately, reverse engineered, to allow
have been shown to increase student academic perform- teachers and students to use technology tools as efficiently
ance, the use of computers often links back to higher as possible. The remainder of this article outlines what this
socio-economic status. re-engineering should involve:
Contradictory results can be found in any domain, but Establish a Vision of How Technology Should Be
for education, which is primarily publicly funded, and for Used: Policymakers and educational designers must have
technology, upon which so much unfulfilled or dashed a vision of what classroom teaching and learning will look
hopes have been placed, such cross-currents fuel techno- like as a result of technology investment and provision.
dissenters’ thesis—educational technology is a disap- This vision building is often the most important—and
pointment and waste of scarce resources. Further, such most overlooked—part of planning for computers in
contradictory results beg the question—why isn’t there schools. Failure to create, articulate and accommodate a
more certainty around the relationship between technol- common vision predictably results in technology projects
ogy and learning? Why after three decades of computers that meander or sputter toward an unanticipated and
in schools, two decades of the Internet in schools, and a unwelcome end. A well-defined and clearly articulated
decade after uptake of mobile learning devices for educa- vision developed by all actors in the education system
tion do we not have greater clarity about technology’s provides coherence to the project, serving as the organiz-
impact on student learning? ing framework within which all goals, actions, infrastruc-
ture, and activities can be developed, and results evalu-

iJET – Volume 8, Issue 1, March 2013 41

 PAPER
SUCCESS, FAILURE OR NO SIGNIFICANT DIFFERENCE: CHARTING A COURSE FOR SUCCESSFUL EDUCATIONAL TECHNOLOGY INTEGRATION

ated. The process of creating a common vision can also Create teacher standards that promote the adoption
help to build engagement and commitment among educa- of learner-centered instructional practices supported
tion stakeholders [33]. by technology: In many countries, there is often no
Develop a Shared Language about Teaching, Learn- compelling reason for teachers to change their instruc-
ing and Technology: Discussions about optimal practices tional practice and/or use computers. On the student-
around classroom technology often resemble Shaw’s wry performance side, as noted above, the curriculum, content,
observation about the English and Americans divided by a and most important, the assessment system, overwhelm-
common language. Terms such as “Information and ingly favor traditional, teacher-centered, fact-based, rote
Communications Technologies” have different meanings instruction. On the teacher-performance side, many
to different stakeholders (They even have different mean- nations, states or provinces often have no standards
ings to the same stakeholders.). Essential instructional against which to assess teachers’ instructional practice and
constructs, such as “learner-centered instruction,” are use of computers as part of classroom instruction; no
often incompletely understood by policymakers and indicators that evaluate the impacts of the adoption of new
teachers. As such there is often a divergence of under- techniques and tools; no mechanisms through which to
standing, and even more critically, incongruent philoso- tailor ongoing professional development inputs; and no
phies of implementation among actors within the educa- coherent framework for the provision of mentoring and
tional system. As a result, profoundly central requisites for support to teachers grappling with the crucible of change
effective technology use, concepts like “integration” or prompted by new pedagogy (learner-centered instruction),
“higher-order thinking,” become clichés and their imple- new tools (computers) and high expectations. Oftentimes,
mentation uneven or superficial [34]. teachers who change their practice do so of their own
volition and personal force of will. They are but a small
Part of establishing a common vision around how
teachers can teach with technology to promote the kind of portion of the overall teaching force (according to change
student learning discussed earlier includes the develop- literature, about 2.5 percent of any population [40]) and
ment of a common language, with shared definitions, not surprisingly, the persistence of those changes typically
standards, levels and outcomes. One advantage of this is attenuates over time.
that by thinking through what terminology means, we can Focus on teacher professional development and
begin to think in terms of levels of use and desired out- support capacity: Many teachers in poor nations and in
comes. underserved regions have weak content and instructional
skills. Yet, implicitly, the fantasy that learning how to use
Align curriculum, instruction, technology and as-
sessment: Content, curriculum, instruction and assess- a computer will somehow transform poor teachers into
ment cumulatively and inextricably drive teacher practice. highly skilled practitioners persists. For such teachers,
Of these, the assessment system holds the greatest influ- computers do not improve their instructional skills; they
ence since it determines instruction—regulating what, and often exacerbate them by shifting the teacher’s efforts
most importantly how, concepts get taught. As noted from instruction and classroom management to technol-
earlier in this paper, there is sufficient research suggesting ogy operations and by diverting the attention of the
that technology yields the greatest learning benefits when teacher education system from improving teachers’
it is used in learner-centered ways [35] [36] [37] [38]. instructional and content skills to teaching teachers how to
There is also abundant research documenting that the use computers.
skills that educationally sound uses of computers can Educational research is unambiguous on this point—
promote cannot be adequately measured by most national student learning is inextricably linked to teacher quality.
examinations [39]. Teachers’ preparation in content and pedagogy are associ-
National examination systems within which teachers— ated with instructional practices and quality, which in turn
particularly middle and secondary school teachers— influence student achievement [41] [42] [43]. Therefore
operate result in the implausibility, if not the impossibility, any school-based technology initiative, particularly in
of utilizing technology as part of learner-centered instruc- contexts where teachers have received inadequate in-
service formation, must focus on improving teachers’
tion, higher-order thinking or project-based learning. Such
instructional approaches are time intensive and written content, instructional and assessment skills and help
teachers identify how computers can or cannot support
examinations may not measure the knowledge and skills
promoted by such approaches. But unless these core these domains.
components—how the curriculum is structured; what Educational research is similarly unambiguous on qual-
content is taught; what level of learning (memorization vs. ity teacher professional development. It must be long-
analysis of content) is cultivated; and what constitutes term, intensive, ongoing, linked to actual teacher class-
“learning” and how it is assessed, computer technology room practice and is far more effective if school-based
will continue be used in the most perfunctory of ways— versus centralized [44] [45]. Most critically, teacher
high-tech tools that support low-level learning. professional development must include an extensive
To support the types of instructional changes promoted ongoing support system. “Support” includes not simply
technical support (like electricity, adequate bandwidth and
by learner-centered technology use, policy makers and
implementers must reform all four of these components functioning computers) but administrative support (school
principals and education inspectors who understand the
purposefully and simultaneously. To focus on one to the
exclusion of the others dilutes the possibility of meaning- new instructional methods necessary so teachers can
adequately integrate computers into content areas); mate-
ful use of technology. It creates the sort of practice-based
tensions which teachers often feel when asked to use rial support (paper, chairs, teaching and learning materi-
als); time; emotional support by school leaders and above
technology in higher-order ways within a curriculum and
examination system that focuses on lower-order skills. all, ongoing and consistent instructional support from a
knowledgeable and caring follow-up person, head teacher,

42 http://www.i-jet.org
 PAPER
SUCCESS, FAILURE OR NO SIGNIFICANT DIFFERENCE: CHARTING A COURSE FOR SUCCESSFUL EDUCATIONAL TECHNOLOGY INTEGRATION

coach or mentor). This is often the most expensive portion as “outcomes” and “impact.” Outcomes are proximal
of school-based technology investments but without this, changes—intermediate effects on participants at an
the returns on investment in educational technology are individual or group level. Impact, in contrast, is distal.
negligible [46] [47]. Impact deals with longer-term changes where unit of
Secure principal involvement: Leadership is a critical analysis is the school or district. Causes aside, the fact
ingredient in school-based change. School leaders/ princi- remains that few national or international evaluations
pals establish the school climate; make decisions about the focus on any sort of meaningful impact.
values and “infrastructure” of the school; and can deter- Fourth, not only are there are no internationally compa-
mine when and by whom computers are used in schools. rable standards by which to measure impact of computers
The connection between supportive and facilitative on student learning, there are often no local standards by
leadership and implementation of innovations is well which to do the same. It is often therefore impossible, or
established in educational research [48]. meaningless, to compare results from one classroom
Just as teachers need to shift from being purveyors of technology program to that of another in a different
information to facilitators of learning in a new educational geographic location or even among schools in the same
paradigm, principals to must move from being purely location.
administrators to becoming instructional leaders. This is a Finally, impact evaluations are the most useful evalua-
role for which most principals are ill-prepared and ill- tions in gauging the effectiveness or lack thereof of the
equipped. Any technology initiative focused on demon- relationship between instruction, technology use and
strating tangible improvements in classroom practice must student learning. But change is not a linear or direct or
include principals – both in the same types of professional immediate process. Impact takes years to accrue but many
development teachers undergo and in their own profes- technology initiatives are short lived (1-5 years). It is
sional development, i.e., focused on the induction and difficult, in many cases, impossible, to measure impact
support required to institutionalize change at the school- after such a short amount of time—akin to a doctor
level. This way, the principal is able to experience and assigning a ten-day regimen of antibiotics and then assess-
understand the new practices, and is also equipped with ing their impact after day three. In reality, many “impact”
additional skills to carry out, support, monitor and evalu- evaluations are conducted in programs that are not mature
ate change. This can ensure a greater degree of alignment enough to be evaluated; many impact evaluations fail to
between the objectives of the technology initiative and the measure impact, measuring instead intermediate effects;
principal’s goals and priorities. and as such many impact evaluations tell only part of the
Restructure technology evaluations: There are pres- story or worse, state that something is or is not working
ently a number of weaknesses with the current system of when in fact it may not have had a chance to fully mature.
evaluating educational technology initiatives. First, The evaluation system for measuring technology’s im-
program evaluations are notoriously tricky affairs, particu- pact on student learning needs reform. It needs common
larly in education and particularly in contexts where standards of measurement; longer timelines for implemen-
people may be ill acquainted with evaluations. As Dede et tation and evaluation; clarity in evaluation-related termi-
al. note, it is exceedingly difficult to assess the impact of nology; a mixture of quantitative and qualitative measures
any innovation within school settings: so we understand the “what” and “why” of a technology
Assessing “impact” (the degree of transformation in innovation; and an underlying awareness that measuring
practice) and “reach” (the number of teachers and impact in school settings, particularly using quantitative
organizations influenced) are important, but complicated. methods is often a tricky proposition.
Often, within the complexity of educational settings, where Put computers in classrooms: One of the reasons
multiple school change and … initiatives may be under- computers yield such disappointing results, particularly in
way simultaneously and students move from teacher to non-industrialized countries, is that students (and teachers)
teacher, it can be difficult to isolate and attribute the can’t get their hands on them. There still remains the
contribution of one … program on a teacher’s develop- belief that computers are a “public good” that must be
ment, and even more difficult to gauge the effect … on housed in a public space—the computer lab. Invariably
student achievement or understanding [49] because they are divorced from classroom instruction,
Next, because of these difficulties, many government these “public goods” remain under-used. Invariably
and aid agencies often ask for evaluations that focus on because these public goods are owned by no one in
measuring inputs (the number of computers delivered, particular, they suffer from the “tragedy of the com-
number of teachers trained) or outputs (number of stu- mons”—ill-maintained, discarded or damaged—
dents who can use Excel). While such evaluations provide administrative decisions that unwittingly confer on com-
a mechanical overview of what occurred, they fail to puters the stigma of complexity and unreliability.
measure impact—why and how an outcome occurred and Computers belong in classrooms. This is where students
the depth and life-cycle of the occurrence. Because are and this is where teaching and learning occur. This
evaluation budgets tend to be far smaller than implemen- statement does not argue for widespread adoption of 1:1
tation budgets, a classroom technology program may computing initiatives, about which research is still inclu-
expend its entire budget on tracking inputs with no larger sive [50] [51]. Rather, with mobile laptop carts, computers
examination of technology’s effects—another missed can be used by students as they study math or science and
opportunity in understanding the benefits (or lack thereof) can be shared equally by all teachers. In 2009-2010,
of educational technology and an omission that mislead- Education Development Center (EDC) provided one
ingly suggests that there may be no impact to measure. laptop to 60 schools in six Indonesian provinces. Technol-
Third, there is often confusion about evaluation related ogy coaches taught teachers how to teach in learner-
terminology, in particular, confusion between terms such centered ways using this one computer. These profes-

iJET – Volume 8, Issue 1, March 2013 43

 PAPER
SUCCESS, FAILURE OR NO SIGNIFICANT DIFFERENCE: CHARTING A COURSE FOR SUCCESSFUL EDUCATIONAL TECHNOLOGY INTEGRATION

sional development sessions did not focus on teaching the REFERENCES

teacher how to use technology; they focused on helping [1] L. Cuban, Oversold and Underused: Computers in the Classroom.
the teacher set up collaborative activities in which students Cambridge, MA: Harvard University Press, 2002.
worked together using their one computer as part of [2] N. Postman, Technopoly: The Surrender of Culture to Technology.
classroom learning. Ninety-eight percent of the 300 New York: Alfred Knopf, 1992.
teachers in the program utilized the one computer in [3] Partnership for 21st Century Skills, 2012.
learner-centered ways and 100 percent of teachers re- [4] R. Kozma, “Technology in Education: Against the Motion,” [The
ported that students showed increased technology profi- Economist Debates] Economist, 2007.
ciency, academic achievement and improved behavior [5] J. Piaget, The Moral Judgment of the Child, London: Free Press,
[52]. Not one computer was stolen or damaged. Children 1932.
learned technology without IT classes. No labs were [6] B.S. Bloom, Taxonomy of Educational Objectives, Handbook I:
needed and little extra space required. The Cognitive Domain, New York: David McKay Co., Inc., 1956.
[7] D.H Jonassen, Computers in the classroom: Mindtools for Critical
Thinking. Columbus, OH: Merrill/Prentice-Hall, 1996.
VII. CONCLUSION
[8] J. Kulik, “Effects of using instructional technology in elementary
Techno-enthusiasts are correct in their underlying and secondary schools: What controlled evaluation studies say,”
premise about technology—computers belong in schools. Arlington, VA: SRI International, 2003.
Part of the mission of education is to prepare students for [9] Center for Research and Innovation, “Are the new millennium
the world of work. The ability to use a computer has learners making the grade? Technology use and educational per-
become basic 21st century literacy, like reading, writing formance in PISA,” Paris: Organization of Economic Cooperation
and Development, 2010.
and numeracy. For school leavers in Mumbai, Delhi or
[10] Commission of the European Communities, “The use of ICT to
Kolkata to get a decently remunerated job, they must support innovation and lifeline learning for all: A report on pro-
know how to use a computer. Students hoping to receive a gress,” Brussels: Author, September 10, 2008.
university degree must be able to use a computer. Since [11] J. Kulik, “Effects of using instructional technology in elementary
private ownership of computers is low in many non- and secondary schools: What controlled evaluation studies say,”
industrialized countries, schools must help students Arlington, VA: SRI International, 2003, p. 60.
acquire these necessary educational and vocational skills. [12] Center for Research and Innovation, “Are the new millennium
Techno-enthusiasts are also correct in arguing that learners making the grade? Technology use and educational per-
formance in PISA,” Paris: Organization of Economic Cooperation
technology provides important educational benefits, and Development, 2010, p. 124.
debates about academic performance aside. Specifically, [13] J. Kulik, “Effects of using instructional technology in elementary
automation, access and equivalence have revolutionized and secondary schools: What controlled evaluation studies say,”
education as they have other domains like agriculture and Arlington, VA: SRI International, 2003.
health. Students can take a final examination online and [14] M. Russell and T. Plati, “Does it matter with what I write?
receive assessment results instantly. Teachers on remote Comparing performance on paper, computer and portable writing
Indonesian islands with no access to professional devel- devices,” Current Issues in Education [On-line], 5 (4), May 29,
opment can participate in professional learning opportuni- 2012. Retrieved November 25, 2012 from
http://cie.asu.edu/volume5/number4/
ties via online courses and collaborate with other teachers
[15] Ibid.
across the globe. Teachers can compensate for poor,
outdated or missing materials through Internet resources. [16] J. Kulik, “Effects of using instructional technology in elementary
and secondary schools: What controlled evaluation studies say,”
Students who find that face-to-face school settings offer Arlington, VA: SRI International, 2003.
an education that is meaningless and irrelevant can opt for [17] Commission of the European Communities, “The use of ICT to
a more meaningful education and gain formal accredita- support innovation and lifeline learning for all: A report on pro-
tion through online programs such as the UK’s gress,” Brussels: Author, September 10, 2008, p. 8
Notschool.net or through any number of online high [18] D. Coates and B.R. Humphreys, (2004). “ ‘No significant
school programs in the US. These are certainly paradigm- distance’ between face-to-face and online instruction: Evidence
changing accomplishments. from principles of economics,” In Economics of Education Re-
view, vol. 23, (6), pp. 533-546, 2004. http://dx.doi.org/10.1016/
But techno-dissenters are correct in their underlying j.econedurev.2004.02.002
premise—technology has more often than not been used [19] J. Angrist and V. Lavy, “New evidence on classroom computers
poorly in schools, particularly in low-resource areas—at and pupil learning,” In Economic Journal, vol. 112, pp. 735-765,
the expense of more necessary interventions (like upgrad- 2002. http://dx.doi.org/10.1111/1468-0297.00068
ing teacher qualifications). The introduction of computers [20] T. Fuchs, and L. Wößmann, L. (2005). “Computers and student
in schools has often been the result of priorities that have learning: Bivariate and multivariate evidence on the availability
little to do with the needs or readiness of teachers and and use of computers at home and at school,” Institute for Eco-
nomic Research at the University of Munich Working Paper,
students. The creation of learning environments that 2005, p. 1.
remain at the heart of why we use technology and for
[21] Center for Research and Innovation, “Are the new millennium
which we strive cannot be attained by technology alone, learners making the grade? Technology use and educational per-
though technology can aid in this endeavor. Such learning formance in PISA,” Paris: Organization of Economic Cooperation
environments cannot be attained with technical solutions, and Development, 2010, p. 103.
though technical assistance is part of the solution. Rather [22] Ibid, p. 133.
they can begin to be attained by strengthening all compo- [23] Ibid, p. 93.
nents of the educational system—vision, teachers’ skills, [24] Ibid, p. 50.
assessment, evaluation and access— so that computers [25] C. Cavanaugh, “The effectiveness of interactive distance educa-
can be integrated into an enabling, versus disenabling, tion technologies in K–12 learning: A meta-analysis,” Interna-
environment. Once this occurs, we may truly be in a tional Journal of Educational Telecommunications, vol. 7(1), pp.
position to answer the question, Does technology help or 73–78, 2001.
hinder student learning?

44 http://www.i-jet.org
 PAPER
SUCCESS, FAILURE OR NO SIGNIFICANT DIFFERENCE: CHARTING A COURSE FOR SUCCESSFUL EDUCATIONAL TECHNOLOGY INTEGRATION

[26] M. Shachar, and Y. Neumann, “Differences between traditional November 25, 2012 from
and distance education academic performances: A meta-analytic http://edpolicy.stanford.edu/publications/pubs/187
approach,” International Review of Research in Open and Dis- [42] Organization for Economic Co-operation and Development,
tance Education, [Online], 4(2), 2003. Retrieved November 25, Teachers’ Professional Development: Europe in International
2012 from http://www.irrodl.org/index.php/irrodl/issue/view/16 Comparison. An Analysis of Teachers’ Professional Development
[27] C. Ungerleider and T. Burns, “A systematic review of the Based on the OECD’s Teaching and Learning International Sur-
effectiveness and efficiency of networked ICT in education: A vey (TALIS), Paris: Author, 2004.
state of the field report,” Council of Ministers Canada and Indus- [43] L. Darling-Hammond and J. Bransford, Preparing Teachers for a
try Canada. Ottawa: Industry Canada, 2003. Changing World: What Teachers Should Learn and Be Able to
[28] V. Spezia, Does computer use increase educational achievement? Do, San Francisco, CA: Jossey Bass, 2005.
Student-level evidence from PISA. [Online], 2003. Retrieved No- [44] D. Sparks and Loucks-Horsley, “Five models of staff develop-
vember 25, 2012 from ment,” In Journal of Staff Development, vol. 10, (4), Fall 1998.
http://www.pcb.ub.edu/xreap/workshop/downloads/papers/avaluat
[45] L. Darling-Hammond, “Professional learning in the learning
s/finals/spiezia.pdf
profession: A status report on teacher professional development in
[29] C. Harrison, et al., ImpaCT2: the impact of information and the United States and abroad,” [PDF document], 2008. Retrieved
communication technologies on pupil learning and attainment. November 25, 2012 from
Coventry: BECTA/London: DfES, 2002. http://edpolicy.stanford.edu/publications/pubs/187
[30] Commission of the European Communities, “The use of ICT to [46] E. Gaible and M. Burns, Using technology to train teachers:
support innovation and lifeline learning for all: A report on pro- Appropriate uses of ICTs for professional development. [PDF
gress,” Brussels: Author, September 10, 2008. document].Washington, DC: World Bank, 2007. Retrieved No-
[31] Center for Research and Innovation, Are the new millennium vember 25, 2012 from
learners making the grade? Technology use and educational per- http://www.infodev.org/en/Publication.13.html
formance in PISA, Paris: Organization of Economic Cooperation [47] M. Burns, Distance Education for Teacher Training: Modes,
and Development, 2010. Models and Methods, Washington, DC: Education Development
[32] O. Valiente, “1-1 in Education: Current Practice, International Center, 2011. Retrieved November 25, 2012 from
Comparative Research Evidence and Policy Implications,”OECD http://go.edc.org/07xd
Education Working Papers, No. 44, OECD Publishing, 2010. Re- [48] T. McCann, L. Johannessen, and B. Ricca, Responding to new
trieved November 25, 2012 from http://dx.doi.org/10.1787/ teachers’ concerns. Educational Leadership, vol. 62(8), 30-34,
5kmjzwfl9vr2-en 2005.
[33] R.B. Kozma, “Comparative analysis of policies for ICT in [49] C. Dede, L. Breit, D. Jass Ketelhut, E. McCloskey and P. White-
education,” In J. Voogt & G. Knezek. house, An overview of current findings from empirical research on
International Handbook on Information Technology in Education, online teacher professional development. Cambridge, MA: Har-
pp.1083-1091. New York, NY: Springer, 2005. vard Graduate School of Education, 2005, p. 5.
[34] M. Burns, “A means to an end: Technology integration,” Thresh- [50] O. Valiente, “1-1 in Education: Current Practice, International
old: Cable in the Classroom, September 2007. Comparative Research Evidence and Policy Implications,”OECD
[35] L. Darling-Hammond and J. Bransford, Preparing Teachers for a Education Working Papers, No. 44, OECD Publishing, 2010. Re-
Changing World: What Teachers Should Learn and Be Able to trieved November 25, 2012 from http://dx.doi.org/10.1787/
Do, San Francisco, CA: Jossey Bass, 2005. 5kmjzwfl9vr2-en
[36] Organization for Economic Co-operation and Development, [51] Texas Center for Educational Research, Evaluation of the Texas
Teachers’ Professional Development: Europe in International Technology Immersion Pilot, First Year Results, Austin, TX: Au-
Comparison. An Analysis of Teachers’ Professional Development thor, April 2006.
Based on the OECD’s Teaching and Learning International Sur- [52] J. Ho and M. Burns, Distance education pilots for Indonesia: An
vey (TALIS), Paris: Author, 2004. evaluation of pilot program 1—Decentralized Basic Education 2
[37] D.H Jonassen, Computers in the classroom: Mindtools for Critical (DBE 2). Evaluation report submitted to the United States Agency
Thinking. Columbus, OH: Merrill/Prentice-Hall, 1996. for International Development. Washington, DC: Education De-
velopment Center, Inc., August 2010.
[38] M. Boethel, and K.V. Dimock, Constructing knowledge with
technology: An overview of the literature, Austin, TX: Southwest
Educational Development Laboratory, 1999. AUTHOR
[39] Organization for Economic Co-operation and Development, Mary Burns is a senior technology specialist at the
Teachers’ Professional Development: Europe in International
Comparison. An Analysis of Teachers’ Professional Development
Education Development Center (EDC) in Waltham, MA,
Based on the OECD’s Teaching and Learning International Sur- USA. She develops and design national educational
vey (TALIS), Paris: Author, 2004. technology plans and educational technology programs
[40] E. Rogers, Diffusion of Innovations, 5th edition, New York: Free across the globe (email: [email protected])
Press, 1995.
Manuscript received 27 November 2012. Published as resubmitted by
[41] L. Darling-Hammond, “Professional learning in the learning the author 27 February 2013.
profession: A status report on teacher professional development in
the United States and abroad,” [PDF document], 2008. Retrieved

iJET – Volume 8, Issue 1, March 2013 45

View publication stats