Computers & Education 63 (2013) 285–305

Contents lists available at SciVerse ScienceDirect

Computers & Education

journal homepage: www.elsevier.com/locate/compedu

A blended learning environment for individualized English listening

and speaking integrating critical thinking
Ya-Ting Carolyn Yang*, Ya-Chin Chuang, Lung-Yu Li, Shin-Shang Tseng
Institute of Education & Center for Teacher Education, National Cheng Kung University, No.1, University Road, Tainan City 701, Taiwan, ROC

a r t i c l e i n f o a b s t r a c t

Article history: Critical thinking (CT) and English communication are recognized as two essential 21st century compe-
Received 28 October 2012 tencies. To equip students with these competencies and respond to the challenges of global competition,
Received in revised form educational technology is being developed to enhance teaching and learning. This study examined the
13 December 2012
effectiveness of integrating CT into individualized English listening and speaking instruction using
Accepted 16 December 2012
Moodle, a virtual learning environment. Individualized instruction was designed with three key
elements, namely proficiency level grouping, individualized instructional strategies and materials, and
Keywords:
individualized feedback. Participants were 83 students enrolled in a semester-long general education
Applications in subject areas
Computer-mediated communication course at a large university in Taiwan. The four dependent measures were CT skills (CTS), CT dispositions
Distributed learning environments (CTD), English listening, and speaking proficiency. Results from the one-group pretest–posttest design
Teaching/learning strategies were evaluated by paired t-tests and a mixed design ANCOVA (analysis of covariance) in order to identify
any statistically significant improvements following the intervention. The results of the study showed
that learners participating in the treatment significantly improved in terms of English listening and
speaking, as well as on all CTS subscales, with little change in CTD, apart from significant improvement
on the subscale of open-mindedness. Limitations and suggestions for successful online CT-integrated
instruction and implications for future research are provided.
Ó 2012 Elsevier Ltd. All rights reserved.

1. Introduction

Teachers and researchers are facing increasing pressure to promote student acquisition of “21st century skills” (Partnership for 21st
Century Skills, 2011), which include communicative competence in local and international languages, higher order thinking abilities,
such as critical thinking (CT), and proficiency in information communication technology. Both challenges and opportunities emerge from
a focus on 21st century skills, often related to the role of technology in evolving learner characteristics and learning environments.
In the 21st century, characterized by increased globalization, competence in international languages, such as English, is invaluable. In
Taiwan, and many other nations where English is a foreign language, students’ listening and speaking skills are often weaker than their
writing and reading skills. According to Yang, Gamble, and Tang (2012), who studied the acquisition of oral proficiency skills with Taiwanese
undergraduate students, this may be due to the following three reasons. First, there is a lack of an authentic English language environment
for English oral communication. Second, the prevailing pedagogy for English learning emphasizes examinations as a primary method of
assessment, focusing on vocabulary, grammar, reading, and comprehension quizzes. As a result, teacher-centered and rote instructional
methods often provide little instructive feedback to students and tend to emphasize literacy skills over oral communication. Third, there is
considerable variance in English proficiency among students, creating an inequality of opportunities for engaging in classroom learning and
difficulties for many students due to a mismatch between their abilities and the course content. Overcoming these obstacles to English
listening and speaking proficiency is the first motivation of our research.
Critical thinking is another key 21st century higher order thinking skills related to success in both academic and employment endeavors.
Due to the influence of a collectivist-oriented culture and a limited view of the learner’s role in education, students in Taiwan have not
traditionally been challenged to think critically and reflectively (McBride, Xiang, Wittenburg, & Shen, 2002). However, a current climate of

* Corresponding author. Tel.: þ886 6 2757575x56230; fax: þ886 6 2766493.

E-mail address: [email protected] (Y.-T.C. Yang).

0360-1315/$ – see front matter Ó 2012 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.compedu.2012.12.012
286 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

educational transformation has fostered an increasingly proactive orientation toward curriculum reform and innovative teaching and
learning. This positive attention and generous governmental funding for programs and initiatives seeking to incorporate higher order
thinking skills at all levels of education has created key opportunities for teaching and research (e.g., Yang, 2008; Yeh, 2012). Effective
integration of critical thinking instruction in higher education is, therefore, the second motivation of this research.

1.1. Enhancing English listening and speaking

English is a compulsory foreign language for higher education in Asia, including Taiwan, due to a growing awareness that learners will
require English language skills in order to gain competitive advantage in the workplace of the future (Nunan, 2003). Since English is the most
used language among Internet usersd565 million people, or 27% of all Internet users, use English for online communication according to
Internet World Stats (2011)dEnglish language proficiency allows users to connect to others around the world and access a vast storehouse
of knowledge available online.
Given the importance of English language skills in the 21st century, standardized tests for English proficiency are often required for job
placement and matriculation from institutes of higher learning. The TOEICÒ test (published by Educational Testing Service) is one such
internationally recognized test, which defines specific skills for assessing listening and speaking. For listening comprehension, participants
respond to questions of four types: (a) photo descriptions, (b) selecting appropriate responses to questions, (c) questions based on
conversations, and (d) questions based on short talks. Speaking tasks evaluated by the TOEICÒ include criteria for pronunciation, intonation,
grammar, vocabulary, cohesion, relevance, and completeness. These speaking tasks include reading text aloud, describing pictures,
responding to questions, proposing solutions, and expressing an opinion.
Research into effective English listening and speaking instruction for EFL learners can be useful in addressing elements of the three key
barriers to learning discussed above, namely lack of authentic interaction in the target language, an emphasis on teacher-centered
instruction, and the difficulties posed by learner heterogeneity. Strategies for supporting authentic English oral communication, learner-
centered pedagogy, and appropriate scaffolding of EFL learners have received a great deal of attention from researchers. In terms of
authentic materials and activities, for example, Yang (2002) found that the teaching materials used in Taiwan were often boring and tedious,
while Wu (2009) further pointed out that an authentic learning environment and learning materials, such as audio/video materials (Larsen-
Freeman, 2000), are essential factors for successfully acquiring the English language in Taiwan. In fact, the goal of authentic communication
is central to English listening and speaking because the very nature and purpose of language is communication (Hadley, 1993; Richards &
Rogers, 2003).
Student-centered instruction is also key to the acquisition of English listening and speaking skills (Yu & Wang, 2009). In contrast to
lecture-based instruction, technology enhanced learning has been proposed as an optimal means for providing students with an envi-
ronment wherein they simultaneously express their ideas, engage in conversation, acquire new knowledge, and stimulate thinking (Khan,
1997). Online learning, or e-learning, has also transformed the nature of English listening and speaking instruction, allowing updating,
storing, retrieving, and sharing of instructional information online (Rosenberg, 2001). Collaborative learning has long been recognized as
a positive language teaching approach in both physical face-to-face classrooms (e.g., Kim, 2008) and computer assisted language learning
(e.g., Lund, 2008). As a result, students show a positive attitude toward learning environments which incorporate computers and collab-
orative learning (AbuSeileek, 2007).
Finally, addressing learners’ needs is one step toward overcoming barriers to English oral communication among English language
learners. For effective language instruction, teachers should allow students sufficient time and flexibility through strategies such as grouping
students according to their learning needs (Larsen-Freeman, 2000). Feedback is also key to assisting learners in language acquisition. Using
an audio blog to assist English speaking and listening, Hsu, Wang, and Comac (2008) found that this format made it possible to provide
individual feedbacks with a positive effect on learners’ speaking and listening.

1.2. Fostering critical thinking

Critical thinking (CT) refers to the process by which one thinks, discovers problems, looks for related information, and makes decisions in
order to solve problems (Bean, 2011; Fisher, 2001). CT is a cognitive strategy influencing learning (Pintrich, Smith, Garcia, & McKeachie,
1991) and is essential in helping improve learners’ leadership abilities, decision-making, critical judgment, and a competitive advantage
for success in the international job market (Partnership for 21st Century Skills, 2011). For contemporary learners, who increasingly rely upon
online sources of information for their studies, aptitudes and cognitive strategies for critically sourcing, evaluating, and applying infor-
mation are indispensable (Barak, Ben-Chaim, & Zoller, 2007; Wen & Shih, 2008). Cultivating learners’ CT, therefore, is one of the highest
priorities for higher education (Halpern, 1999; Halx & Reybold, 2005; Yang & Chou, 2008) and is receiving considerable attention from
researchers in the field of instructional design (Mok, 2010; Nagappan, 2001).
Generally speaking, CT consists of two main elements: a cognitive component, CT skills (CTS), and an affective component, CT disposition
(CTD). While CTS refers to the cognitive forces behind problem-solving and decision-making, CTD refers to the internal motivation to engage
in critical thinking. On the basis of a systematic inquiry into the construct of CTS and CTD adopting the Delphi Method, a set of six cognitive
skills were defined: interpretation, analysis, evaluation, explanation, inference, and self-regulation (Facione, 2007). CTD, additionally,
consists of seven conceptual elements: inquisitiveness, truth-seeking, analyticity, systematicity, CT self-confidence, open-mindedness, and
cognitive maturity. For definitions of these skills and dispositions, see Table 1.
Studies from undergraduate EFL classroom settings have advocated strategies for fostering CT which emphasize the core concepts of
authenticity, learner-centered instruction, and scaffolding of learner needs. In terms of authentic content, Florez (2000) advises a gradual
scaffolding of materials, working from simpler to more complex cases, with topics that are relevant and interesting, making use of authentic
texts. The role of content in CT-based instruction is crucial, since learners must first have a grasp of the topic before developing the
confidence and capacity to think critically about related issues. Thus, content-based language education stems from student interaction with
authentic texts which offers experience in using language and developing skills necessary for other courses (Dantas-Whitney, 2002), an
important step toward transfer of learning.
 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 287

Table 1
Definition of elements of critical thinking skills and dispositions (Facione, 1990a, 2007).

Factor Description
CTS Analysis Determining significance, interpreting meaning and detecting possible inferential relationships
Inference Bringing together all the various elements needed to draw reasonable conclusions, and forming conjectures and hypotheses
Evaluation Assessing the credibility of statements and the strength of arguments.
Justifying one’s reasoning based on relevant evidence, concepts, methods or standards
Deductive The assumed truth of the set of premises purportedly necessitates the truth of the conclusion.
reasoning
Inductive reasoning An argument’s conclusion is apparently warranted, but not necessitated, by the assumed truth of its premises
CTD Analyticity Demanding the application of reason and evidence and inclined to anticipate consequences
Open-mindedness Tolerance of divergent views and willingness to seriously entertain alternatives. Self-monitors for possible bias
Truth-seeking Courageous desire for the best knowledge, even if such knowledge may fail to support or undermine one’s preconceptions, beliefs or interests
Systematicity Valuing organization, focus and diligence to approach problems at all levels of complexity
Self-confidence Trusting of one’s own reasoning skills and inclined to use these skills, rather than other strategies, to respond to problems
Inquisitiveness Curious and eager to acquire knowledge and to learn explanations even when the applications of the knowledge are not immediately apparent
Maturity Prudence in making, suspending or revising judgment

Student-centered learning strategies are naturally a necessity for the fostering of higher order thinking, since the emphasis is on attitudes
and skills which cannot simply be acquired through passive learning. Several strategies have been successfully implemented for fostering CT,
including debates and discussions. Several authors, such as Benesch (1999) in her meta-analysis of CT in postsecondary second language
classes, mentions the successful use of debates for fostering learners’ critical thinking, findings which have been substantiated by other
researchers (e.g., Kovalik & Kovalik, 2007; Shaaban, 2005). Perspective-taking discussions and sharing of personal experiences and opinions
(Crismore, 2000; Pally, 1997) are also effective means of fostering CT through oral communication for postsecondary and adult learners.
Finally, concerning learners’ needs and characteristics, we must also consider the role of culture and context in terms of CT instruction. In
traditional Confucian education, the role of the learner is considered passive; thus students may not be challenged to think critically and
reflectively. Therefore, CT instruction for Eastern undergraduate students’ may need to be more explicit (Yang & Chou, 2008) in order to
provide a relaxed and warm atmosphere (Chau et al., 2001) and provide sufficient time to think (Yeh, 2004), particularly when required to
do so in a foreign language. Ultimately, CTS and CTD will be fostered through a culture of reasoned and logical thinking, modeling of positive
CT behaviors, opportunities for peer interaction, and a process-based, rather than results-based, mode of reinforcement (Facione, Sanchez,
Facione, & Gainen, 1995; Tishman, Jay, & Perkins, 1993).

1.3. Integrating CT with English listening and speaking instruction

A close relationship exists between language and thinking, and pragmatically, intelligible and comprehensible communication in English
or any other language, requires CT elements such as explanation, analysis, evaluation, and interpretation (Kennedy, Fisher, & Ennis, 1991).
The effective instruction of CT has drawn researchers’ attention for a while, and since 1990s, CT instruction has shifted from being separate
from the instruction of subject matter to being embedded in the subject matter (Helpern, 1997). Several researchers (e.g., Bean, 2011; Kurfiss,
1988) have demonstrated the advantages of using problems, questions, or issues as a source to promote learner motivation and sustained
inquiry. The role of collaboration in CT learning for postsecondary education has also been emphasized (An, 2006; Şendag  & Odabaşɪ, 2009).
While studies have indicated a link between CT and academic performance in subjects such as 6th grade math (Eshel & Kohavi, 2003),
second and third year undergraduate psychology (Phan, 2009), and undergraduate business education (Rodriguez, 2009), the relationship
between CT and English language learning has not drawn serious attention from researchers until quite recently. Among the few scholars
examining the relationship between CT and English language learning, Birjandi and Bagherkazemi (2010) and Ghaemi and Taherian (2011)
evaluated the role which teachers’ CT plays in their teaching of EFL adult learners, while Fahim, Bagherkazemi, and Alemi (2010) found that
adult EFL learners’ reading scores were significantly associated with their CT scores.
Listening and speaking are frequently involved in learners’ acquisition of CT. According to empirical research (Alagozlu, 2007; Chapple &
Curtis, 2000; Davidson & Dunham, 1997), postsecondary English language learners enhance CTS through oral discussions and sharing or
expressing opinions. In conversation classes, for example, Wang (2009) studied 65 freshmen and found that both teachers and learners
require opportunities to express their individual experiences in order to develop the skills of organizing thoughts and engaging in critical
reflection. To implement CT instruction, Bean (2011) lists a variety of ways for CT university instructors to design CT tasks, including whole-
class discussions and in-class debate, which aim not only at creating a collaborative and authentic context-based learning environment, but
also at converting students from passive to active learners. To achieve the same instructional goal, Peterson (2009) proposed an application
of Richard Paul’s (1995) Socratic method, a systematic and multifaceted approach focusing on issues or problems (Paul & Elder, 2007).
Similarly, Golding (2011) stressed the importance of instructors asking and addressing thought-encouraging questions by adopting Socratic
questioning to help students cultivate CT and become critical thinkers.
While such studies have suggested that CTS can be enhanced through English listening and speaking activities, the potential for CT-
integrated English instruction for promoting CTD remains unexamined. Thus, one challenge of this study was to integrate CT into
English listening and speaking instruction and examine the improvement of learners’ CT performance, including both CTS and CTD.

1.4. Individualized instruction

A few instructional strategies for creating authentic and learner-centered environments for language and CT acquisition have been
discussed. However, one remaining difficulty in offering CT-integrated English language instruction to students, (particularly in large,
compulsory classes), is the large degree of variance in proficiency levels among learners. One solution to the problem is individualized
instruction, which takes into account each learner’s characteristics, background, and needs (Recker, Ram, Shikano, Li, & Stasko, 1995). Wang
288 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

and Walberg (1983) offer a widely accepted definition of individualized instruction as “the use of alternative instructional strategies and
school resources to provide learning experiences that meet the different needs of individual learners” (p. 603). According to Walberg (1975),
individualized instruction can be categorized into three elements: selection, enrichment and acceleration. Selection provides learners with
choice in learning content, based on their background, interests, and learning needs. For enrichment, learners are provided with a variety of
teaching materials, which extend beyond the core curriculum. Acceleration refers to learner arrangement of their own learning schedule
based on their individual needs, with periodic assessment and feedback provided to assist learners in evaluating their progress. Overall, the
main goal of individualized instruction is to provide varied activities, a suitable learning environment, and appropriate resources and
technologies to meet a wide variety of student needs.
While individualized instruction, as described above, does address the issue of variance in learner abilities and needs, we are still faced
with the problem of how to incorporate individualized learning in large and highly heterogeneous university classes. After all, in practice, it
can be difficult and time-consuming to implement individualized instruction for each student. Thus, the current study seeks to achieve
a practical and effective form of individualized learning through the following characteristics: proficiency grouping, individualized
instructional materials, and individualized feedback.

1.4.1. Proficiency grouping

To satisfy a variety of learner needs and backgrounds, proficiency in specific skills (i.e., English listening and speaking) was determined to
be key variables for individualized instruction. Researchers (e.g., Gamoran, Nystrand, Berends, & LePore, 1995; Novak, 2010) have generally
suggested grouping learners based on their learning abilities before implementing individualized instruction, although their proposals on
how to group individuals varies. Our solution to this challenge is discussed in the Methods section of this paper.

1.4.2. Individualized instructional strategies and materials

Once learners are grouped according to proficiency, instructional materials can be designed for each group or level (Ireson & Hallam,
2001). In this case, the “selection” element is fixed according to student proficiency levels, but instructional design based on each group
provides opportunities for “enrichment” (through the provision of a variety of level-appropriate learning resources) and “acceleration”
(particularly in the case of self-paced learning through an adaptive learning system).

1.4.3. Individualized feedback

Additionally, individualized instruction includes assessment and feedback mechanisms, critical for enabling “acceleration” (Walberg,
1984). In addition, feedback is fundamental for fostering critical thinking (Tishman & Andrade, 1999) and foreign language learning
(Ferreira & Atkinson, 2009).

1.5. Virtual learning environments for individualized learning

Individualized instruction, with grouping based on proficiency, can address the difficulties faced by learners in classrooms with mixed
ability levels. However, standard, lecture-based instructional approaches will simply not support individualized learning. Alternatively,
online or blended learning may take advantage of virtual learning environments (VLE) designed for achieving individualization by creating
a learner-centered and interactive environment for efficient learning (Karagiannidis, Sampson, & Cardinali, 2001; Shang, Shi, & Chen, 2001).
These online environments overcome the physical and time constraints of a typical classroom, offering learners an individualized and
adaptive learning experience (Meurant, 2010; Shute & Towle, 2003).
As a prime example of a virtual learning environment, Moodle (Modular Object Oriented Dynamic Learning Environment) provides
a variety of features for implementing models of individualized instruction. Martin Dougiamas, Moodle’s lead developer, claims that the
design and development of Moodle was based on social constructivist pedagogy (Dougiamas & Taylor, 2003), emphasizing collaboration and
inquiry and discovery-based learning (Brandl, 2005; Corich, 2005). Language learning requires social interaction between the instructor and
the learner, and among the learners themselves, and the use of a VLE makes it possible for learners to accomplish asynchronous learning
tasks individually or collaboratively, and receive both instructor and peer feedback (Brandl, 2005).
Moodle was selected for the current study not only because it is free and open-source software, but also because it incorporates a variety
of user-friendly functions which allow designers to build custom multiuser digital learning platforms for teaching and learning. As an e-
learning platform, Moodle easily incorporates multimedia content, ideal for language learning and enhancing learner motivation. In
addition, users can access to the Moodle system through any web browser, completing their course work at their own convenience and pace.
The research on extending virtual learning environment functionalities to individualized instruction is still emerging (Al-Ajlan & Zedan,
2008; Hölbl & Welzer, 2010; Wu, 2008), and there is little empirical evidence on the effectiveness of implementing virtual learning envi-
ronments for language learning. To address this research gap, our study examines the impact of an online individualized English virtual
learning environment built on the Moodle system for CT-integrated English language instruction on participants’ English listening and
speaking proficiency.

1.6. Research questions

By examining the effectiveness of integrating CT into individualized English instruction by using a blended learning environment with
both face-to-face and online instruction, the study attempted to answer the following four inter-related questions:

RQ1: Will participants demonstrate improvement in CTS, including analysis, inference, evaluation, induction, and deduction?
RQ2: Will participants demonstrate improvement in CTD, including truth-seeking, open-mindedness, analyticity, systematicity, CT self-
confidence, inquisitiveness, and maturity of judgment?
RQ3: Will participants of all levels (advanced, intermediate, and basic) demonstrate improvement in English listening performance?
RQ4: Will participants of all levels (advanced, intermediate, and basic) demonstrate improvement in English speaking performance?
 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 289

1.7. Conceptual framework

Our study seeks to design a blended learning environment for the effective instruction of critical thinking integrated with English
language learning. As such, we considered the current situation in Taiwan and strove to overcome the three obstacles to the acquisition of
English oral communications skills proposed by Yang et al. (2012), which are also related to the development of critical thinking skills: lack
of authenticity, lack of a student-centered approach, and large variance in learner proficiency and background. Based on an extensive review
of the literature, several key theoretical principles for optimal language and critical thinking instruction can provide potential solutions to
these learning obstacles. Table 2 contains a theory-based framework for addressing these obstacles, and provides examples of practical
applications of these theoretical principles as they relate to the design and implementation of our project.

2. Material and methods

2.1. Participants

Participants were recruited from students enrolled in an elective course entitled “Digital English Learning and Critical Thinking Training”
at a large university in southern Taiwan. A total of 83 students (45 males and 38 females; mean age of 19) from various colleges, including
Engineering, Management, Medicine, Science and Social Science consented to participate in this study. Pretest listening and speaking scores
on the Test of English for International Communication (TOEICÒ) were used to group learners into three proficiency levelsdadvanced,
intermediate, and basic. Participants were assigned according to their pretest scores, based on TEOICÒ guidelines, shown in Table 3.
The research team was led by a professor from the Institute of Education with ten years of experience in CT research and instruction. The
lead researcher served as the main English instructor, with one English native speaking Education doctoral student with seven years of EFL
teaching experience serving as lead teaching assistant, two Education Master’s students serving as teaching assistants, and one post-
doctoral researcher with five years’ experience in building websites performing the role of website developer and administrator. The
team collaboratively designed the listening and speaking course for the three proficiency levels based on an extensive review of the
literature, monitored learner progress, provided assistance through the course website and during office hours, and provided online
individualized feedback. The lead researcher and teaching assistants facilitated CT instruction modeled on Socratic dialogs and cultivated
learners’ CTD during synchronous online discussions. The post-doctoral researcher was responsible for designing and maintaining the
virtual learning environment (i.e., the Moodle system) specifically for this course.

2.2. Research design

The research design is shown in Fig. 1. A pretest–posttest one-group design was adopted in order to address the aforementioned research
questions. It is worth noting that no control group was used. First, the number of participants was limited by the size of the technology-
enhanced classrooms available. Second, a semester-long pilot test demonstrated improvement in both CT and English achievement

Table 2
Conceptual framework linking theory and practice for solving three learning obstacles.

Learning obstacle Theory-based principles Relevant Application to the research design

researchers
Lack of authenticity Authentic and content-based Dantas-Whitney  Source authentic listening materials based on themes applicable to
language learning (2002); student experiences
Florez (2000);
 Incorporate online discussions for authentic interactions in English
Hadley (1993);
Wu (2009);
Yang (2002)
Debates and discussions for CT Benesch (1999);  Design synchronous online discussion and debate activities
Crismore (2000);  Provide opportunities for students to share their experiences and opinions
Pally (1997);
Shaaban (2005)
Close association of language Helpern (1997);  Integrate English language learning with CT instruction
and thought Kennedy et al.  Create online activities which simultaneously foster CT and English
(1991)
oral communication
Lack of student-centered Social constructivism An (2006);  Use collaborative learning activities (such as debates and discussions)
learning  and
Şendag  Scaffold learning through online learning tasks
Odabaşɪ (2009)
CT instruction by direct Marin and  Create weekly tutorials and practice modules highlighting specific
instruction with discussion Halpern (2011); CT skills
Yang and Chou
 Offer in-class modeling and discussion of CT
(2008)
Diversity of learner needs, Individualized instruction Ireson and  Group students by (pretest) proficiency level
abilities, and experience Hallam (2001);  Develop materials based on student needs and proficiency levels
Novak (2010);
(i.e., three levels)
Wang and
Walberg (1983)  Provide individual and immediate feedback
Online or blended learning Brandl (2005);  Design a virtual learning environment for optimally fulfilling the
Meurant (2010); principles and instructional objectives listed above
Shute and Towle
(2003)
290 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

Table 3
TOEICÒ scores of the three level groups in the study.

Listening Speaking

Assignment guidelines Number of participants Assignment guidelines Number of participants

Basic <197 3 0–60 8
Intermediate 198–374 43 60–130 49
Advanced 375–495 37 130–200 26

following CT-infused learning in a technology-enhanced classroom environment (Gamble & Yang, 2010). Thus, for this stage of the research,
which adopted an online, individualized learning platform, it was deemed more ethical to provide an optimal learning environment to all
students registering in both sections of the class. Finally, the special class designed for this study was offered as a general studies course,
requiring the evaluation and approval of a university-wide curriculum review committee.
Although no control group was used, the one-group pretest–posttest design does allow for control of many variables through the use of
a pretest, wherein the effect is measured as the difference between posttest and pretest measures. Problematically, the lack of a control
group makes it difficult to assess the potential influence of extraneous variables such as maturation, instrumentation, and regression
artifacts (Johnson & Christensen, 2008).
Pretests and posttests were held during the first two and last two weeks of the course. During these classes, participants filled out
questionnaires, took the California Critical Thinking Skills Test (CCTST), the California Critical Thinking Disposition Inventory (CCTDI), and
TOEICÒ listening and speaking tests (Form A for the pretest and Form B for the posttest). The remaining eight weeks were allocated for CT-
integrated English listening and speaking sessions (as shown in Table 4).

2.3. Dependent variables and instruments

The four dependent variable measures included CTS, CTD, and English listening and speaking abilities, measured respectively by the
CCTST, the CCTDI, and TOEICÒ Listening and Speaking tests.

2.3.1. Critical thinking skills

The CCTST (Facione, 1990a; 1992) consists of 34 multiple-choice questions and takes 45 min to complete. The test, suitable for college
students, graduate students, and adult professionals, generates a total score and five subscale scoresdanalysis, evaluation, inference,
deductive reasoning, and inductive reasoning. A full score for the instrument is 34; with scores below 14 indicating basic CTS; scores
between 14 and 20 intermediate CTS; and scores above 20 high CTS. The internal consistency reliability of the published version of the two
parallel alternate forms are KR-20 ¼ .70 (Form A) and KR-20 ¼ .71 (Form B). The Mandarin version of CCTST was used for this study.

2.3.2. Critical thinking dispositions

The CCTDI contains 75 “agree-disagree” Likert type questions and takes about 20 min to complete. Seven categoriesdinquisitiveness,
open-mindedness, systematicity, analyticity, truth-seeking, CT self-confidence, and cognitive maturitydare included in the instrument. The
CCTDI is designed for use with students at the high school level and above. The scores on each subscale range from 5 to 60, with overall
scores ranging from 35 to 420. Scores above 350 reflect a strong disposition toward CT, while scores between 280 and 350 indicate a positive
inclination, and scores below 280 reflect an overall deficiency in CTD (Facione, 1990b). The overall reliability of the CCTDI is Cronbach’s
Alpha ¼ .90. Cronbach’s Alpha for the seven CCTDI subscales’ range from .72 to .80. The Mandarin version of CCTDI was selected for use in
this study.

2.3.3. English oral communication skills

TOEICÒ was used to measure participants’ English listening and speaking proficiency. The test was developed by the Educational Testing
Service (ETS) to assess the English proficiency of non-native speakers. TOEICÒ is one of the most popular and globally recognized tests for
English proficiency and has been used as a standard evaluation tool for business, educational, and governmental organizations. The listening
test consists of multiple-choice questions, while the speaking test questions are based on real-life settings such as business meetings,
interviews, business letters, and travel.

Fig. 1. Research design.

 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 291

Table 4
Course schedule with weekly content and CT-integrated English activities.

Week Module Theme CT direct CT-integrated English activities Targeted skills

instruction
CTS CTD
1, 2 Pretests (CCTST, CCTDI, and TOEICÒ)
3 Listening I Entertainment – Going If P then Q Selecting a movie to watch by analyzing Inference, deduction Systematicity
to the movies the pros and cons of each genre
4 Listening II Entertainment – Sports Statistics and Analyzing the arguments and biases of Deduction, induction, Open-mindedness
on TV syllogism an interview with a young basketball star analysis
5 Speaking I Restaurants – Work Thinking Talk about yourself by using Socratic Analysis, induction CT-confidence,
stimulation Questioning Techniques inquisitiveness
6 Speaking II Restaurants – Dining Post hoc Responding to request for information and Analysis, deduction, Systematicity, truth-
proposing a solution to a problem induction, evaluation seeking
7 Listening III Business – Advertising Analogical Listening to an advertisement and evaluating Analysis, deduction, CT-confidence, open-
arguments its assumptions inference mindedness
8 Listening IV Business – In the office Tautology Listening to an article on management and Analysis, induction, Truth-seeking,
explaining the analogies heard rvaluation inquisitiveness
9 Speaking III Daily life Evidence Expressing and explaining one’s stance on the Evaluation, analysis Inquisitiveness,
statement, “once a cheater, always a cheater” maturity,
10 Speaking IV Daily life Who makes Peer discussion of whether companies should Evaluation, deduction, Truth-seeking,
sense? ban social networking sites at work induction systematicity
11, 12 Posttests (CCTST, CCTDI, and TOEICÒ)

The listening tests, administered at the beginning and at the end of the semester lasted 45 min; types of test questions included
photograph descriptions (10 questions), question-response matching (30 questions), conversations (30 questions in 10 units), and talks (30
questions in 10 units). Based on participant responses, listening comprehension scores are computed on a scale from 5 to maximum of 495.
The number of correct answers to a total of 100 questions was converted to a scaled score based on ETS guidelines. These scaled scores were
then used to determine the proficiency level group assignment for each participant (see Table 3).
The speaking tests, administered at the beginning and at the end of the semester, lasted for 20 min. Six types of test questions included
read aloud texts (2 questions), picture description (1 question), response to questions (3 questions), response to questions using information
provided (3 questions), proposal of a solution (1 question), and expressing an opinion (1 question). The first four types of questions were
scored between 0 and 3, while the last two types of questions were scored on a scale from 0 to 5, based on a scoring rubric provided by ETS.
Based on participant responses, speaking proficiency scores are computed on a scale from 0 to a maximum of 200. Two experienced raters
evaluated test results after a thorough discussion of the official rating rubric. The scoring procedure was verified by an inter-rater reliability
measure, in which the responses from the two raters were compared, resulting in an inter-rater reliability of .91. In terms of rater quali-
fications, the first rater was an English native speaker and PhD candidate in Education, while the second rate held a Master degree in
Teaching English to Speakers of Other Languages.

2.4. Research procedures

The research procedure includes stages for curriculum and online environment design, grouping of students by proficiency level,
implementing designed individualized learning activities (both face-to-face and online) in a blended environment, and providing feedback
on students’ learning progress. These stages of the research and treatment procedures are illustrated in Fig. 2.

2.4.1. Curriculum content and design

Individualized instructional activities were designed based on a review of the literature, briefly overviewed in the Introduction section, and
incorporated eight topics selected from four related themes (Entertainment, Dining, Business Life, and Daily Life). Content was sourced from
the public domain, created by the researchers, or adopted from materials using educational “fair use” guidelines (i.e., less than 10% of any
source was used, content was available online only for enrolled students for the duration of the course, and no profit was derived from the use
of materials). Materials included audio and written samples from materials designed to address daily life content and issues. Materials and
activities were adapted for online instruction according to the proficiency level of each group, meeting the individualized instruction element
of “selection”. In order to allow learners opportunities for “enrichment”, additional audio-visual and text-based materials were compiled.
In order to allow for “acceleration” and mastery-based learning within the constraints of the course syllabus, learners were given one
week to complete each module of the course, but were allowed to complete online course activities at their own pace and were encouraged
to repeat activities until they were confident in their performance. To this end, immediate, automated feedback was provided for online
activities, such as cloze texts, multiple choice questions, and matching activities. Student speaking activities (which were recorded and
uploaded to the course website) were evaluated and detailed feedback was provided by teaching assistants within one week of being posted.
Concerning the key weaknesses of Taiwanese students in terms of listening and speaking, the instructional design involved strategies for
(a) overcoming a lack of authentic English listening and speaking activities, (b) replacing a lecture-style approach with student-centered
learning, and (c) taking into account diversity in learner proficiency when designing course materials. In terms of providing opportunities
for real-life English oral communication, the curriculum adopted several features, including: authentic audio-visual materials from a variety
of sources, providing opportunities for online discussion in an English immersion environment with a native-speaking teaching assistant,
and adopting themes which were better suited to daily life experiences (i.e., entertainment, business settings, and dining experiences). In
order to promote student-centered learning, the course design featured: activities providing immediate feedback to build motivation, an
emphasis on self-paced learning to enhance self-regulated learning, and collaborative activities to foster peer interaction. Additionally, the
online platform allowed each student to create a profile, upload an avatar, indicate their interests and hobbies, and interact with peers and
teaching assistants through the VLE.
292 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

Fig. 2. Research procedures.

Through the use of interesting and relevant content, accessible and flexible learning activities, and an emphasis on collaborative learning
and problem-solving, the CT-integrated online course was designed with the student’s interests, needs, and characteristics in mind. Finally,
in order to alleviate inequalities in listening and speaking opportunities caused by variance in learners’ experience and proficiency in
listening and speaking, participants were assigned to groups based on pretest TOEICÒ scores. Course materials were designed to scaffold
student learning by adjusting lexical difficulty, length, and complexity of materials in order to challenge but not overwhelm learners at the
basic, intermediate, and advanced levels. Furthermore, practice and assessment activities were tailored for each level, and group size for
online discussions were limited to five members, with online activities designed to ensure each participant a relatively equivalent amount of
speaking time.
For CT-integrated English instruction, both direct instruction of CT concepts (using Mandarin and English presentations and in-class
discussions to ensure comprehension) and collaborative online CT activities were designed. In terms of direction instruction and in-class
discussions, themes were designed based on a review of the literature, ensuring the inclusion of each skill and disposition proposed by
the CCTST and CCTDI. As shown in Table 4, topics such as “If P then Q”, “Analogical Arguments”, and “Socratic Questioning”, designed to cover
both CT skills and dispositions. The first four weeks of CT-integrated instruction focused on an introduction to CT concepts and practice of CT
activities with the last four weeks emphasizing CT dispositions and applications to daily life.
Critical-thinking integrated activities utilized the same themes for each English proficiency level, but materials were adapted to allow the
appropriate level of difficulty for each group. In-class discussions were conducted in order to better model CT thinking skills and dispo-
sitions, with weekly CT activities designed to match the listening or speaking topic of that week. As such, collaborative critical thinking
activities such as debates and Socratic questioning were designed based on previously learned content, achieving a “sustained content”
approach necessary for equipping learners with the background necessary for thinking critically (Pally, 1997).

2.4.2. Development of the online platform

For the virtual learning environment, we chose to use Moodle, since it is easy to install, provides multi-language access, and uses
a module approach to construct the website. This modular approach was convenient for the creation of learning strategies and materials
such as audio-visual and digital materials, a variety of online assessment and practice activities, assignment upload applications, online
asynchronous forums, and automated and teacher feedback systems. The VLE was easily accessed by learners (who were grouped by
proficiency level) who were automatically directed to materials and activities fitted to their learning needs. Instructors and teaching
 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 293

assistants had access to revise and upload materials, evaluate student progress, respond to inquiries from participants, and review and
improve online content based on student feedback and learning outcomes.
A post-doc specializing in distance learning was responsible for installing and testing the system. The learning platform server utilized
a Microsoft Windows operating system running an Apache 2.0 server with PHP 5.2.3 and MySQL 5.0.45 database system. Version 1.9.5 of
Moodle was installed using default settings. The course homepage was separated into a left navigation pane (including online users,
a bulletin board, links to online discussion software, an F.A.Q. section, and a contact list for the instructor, web administrator, and teaching
assistants) a right navigation pane (including a digital clock, course calendar, office hours schedule, and supplemental course materials), as
well as the central content pane wherein the main menu for course activities was displayed. Course materials were designed by the
instructor and teaching assistants, who then uploaded these materials to the system platform. A total of three months was spent in
preparing the VLE and uploading the designed course content for each for each of the three proficiency levels. Compiled SCORM course
packages, answer sheets, and hand-outs were uploaded by the Moodle administrator in advance of each week’s instruction, allowing the
course designers to modify content as necessary before allowing students online access.

2.4.3. Treatment procedures

The class was held in a technology-enhanced classroom with individual computer stations for each student, internet access, and an
integrated network for audio-visual presentations. The classroom was booked for two 50-min periods each week, including 20 min of face-
to-face instruction and 80 min of supervised online learning in the classroom. Themes of English listening and speaking instruction included
entertainment (listening), restaurant (speaking), business (listening/speaking), and daily life (speaking), as shown in Table 4. A short
introduction covering technical issues, including how to upload files to Moodle, was included in the first session at the beginning of the
semester. Free computer access was provided for learners who may not have had internet access at home. Based on the three practical
design principles for individualized learning (discussed in Section 1.4), the following treatment elements were included:

2.4.3.1. Proficiency level grouping. Each week, students spent a total of 150 min learning English in a blended environment. After logging in
the Moodle system with account name and password, students were only allowed to view the content at the level corresponding to the
group to which they were assigned. For example, as Fig. 3 shows, “YC Wang” (a pseudonym) was assigned to the intermediate group for
listening and basic group for speaking. Every student completed the learning packages designed specifically for their proficiency level.

Fig. 3. Proficiency level grouping using the Moodle system.

294 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

2.4.3.2. Individualized instructional strategies and materials. Overall, the curriculum included three main parts: (a) English instruction, (b) CT
direct instruction, and (c) CT-integrated English activities. Individualized instructional strategies and materials were designed and employed
for English instruction and CT-integrated English activities for each week of the study. During the Listening II module, for example, the topic
was “Sports on TV.” By comparing the content and learning activities for “Listening C” (i.e., the third set of activities from the online learning
module for Listening instruction), we can observe how instructional strategies and materials were individualized (see Fig. 4). For the basic
group this activity consisted of three sections based on a 111 s, slow-paced listening sample. The first section asked two basic questions
about the topic, while the second section repeated the audio and asked participants to complete a cloze activity using a list of key words. The
third section included five basic comprehension questions based on the same track. For the intermediate group, the activity explored
a similar topic with three listening comprehension questions, ranging in length from 21 to 23 s. The recording was moderately paced and
required listeners to attend to contextual cues to determine the topic of each audio clip. For the advanced group, however, the activity was
based on a 90 s, fast-paced (i.e., at a natural tempo for two native speakers of English engaged in lively conversation) audio recording. Five
listening comprehension questions were designed to (a) differentiate similar sounding words, (b) demonstrate understanding of the story
plot details, and (c) draw inferences from the content.
From this example, we can see how elements such as speaking pace and lexical difficulty of audio content were matched to learner
proficiency levels. As well, practice activities were varied in terms of the amount of scaffolding and support provided in the form of prompts

Fig. 4. Individualized instructional activities for the listening II module.

 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 295

Fig. 4. (continued).

and hints. Some criteria adopted when designing instructional activities for fulfilling learning objectives for the three proficiency levels can
be found in Table 5.
CT direct instruction, however, was not designed with individualization in mind, but involved the entire class. Take, for example, one of
the warm-up activities for CT instruction in the Speaking III module. The activity provided four perspectives on the statement, “once
a cheater, always a cheater,” designed to model an open-minded attitude toward an issue, which (open-mindedness) was one of the foci of
CTD instruction for this module. Participants were then asked to express their own opinions in English by applying the CT skills of analysis
296 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

Table 5
Individualized instructional activities and learning objectives for the three proficiency levels.

Listening Speaking

Instructional activities Learning objectives Instructional activities Learning objectives

Basic U Multiple choice U Understand the content U Focus on pronunciation U Pronounce syllables accurately
U Comparison U Identify main ideas U Form sentences and answer questions U Describe photographs
U Cloze text U Identify details U Group discussions and role plays U Make polite requests
U Checklist U Organize what is heard U Express an idea
Intermediate U Basic activities U Basic objectives U Basic activities U Basic objectives
U Restructuring sentences U Understand non-native speech U Focus on accent U Use proper intonation
U Engage in interviews and speeches U Express an opinion
Advanced U Intermediate activities U Intermediate objectives U Intermediate activities U Intermediate objectives
U Correct mistakes U Identify implied hypotheses U Forming paragraphs U Improve fluency and pace
U Answer questions U Group debates U Provide appropriate content
U Discussion

and evaluation (the foci of the week), recording their opinion as a digital file before uploading it to the Moodle system. The research team
then provided individualized feedback on both speaking (such as pronunciation, fluency, and vocabulary) and CT-related criteria.
Furthermore, online synchronous group discussions were included as extension activities. Each of the four online discussions took place
over two weeks, with learner-led online discussions for the first week and instructor and teacher assistant facilitation provided for the
second week. Individualization was achieved for these activities, as participants were grouped by proficiency level. Each discussion group
(with an average of five members) spent 30 min each week discussing the assigned topic via Skype, a free online conferencing software. For
example, during the Speaking IV module, basic and intermediate group members expressed their opinions on the issue and challenged each
other with questions, while advanced group members used a more formal style of debate. Through these discussion activities, learners
applied what they had learned from both CT-integrated English listening and speaking instruction and direct CT instruction.

2.4.3.3. Individualized feedback. To meet each student’s needs, individualized feedback was integrated into the learning process. The first
type of feedback, as shown in Fig. 5, was provided immediately by the online platform. Students were able to receive feedback (such as
affirmation of correct responses or hints for incorrect responses) immediately after answering a multiple choice question or completing
a cloze activity. Other types of feedback came from the instructor, teacher assistants, and peers, for the weekly homework or asynchronous
discussions. After students uploaded files of their recordings of homework and online discussion assignments, the instructor and teacher
assistant accessed the system, downloaded the files, and provided individualized feedback (as illustrated in Fig. 6) for oral communication
and CT-related assessment criteria. Feedback included both text and audio files for students to improve their intonation, pronunciation, and
fluency.

3. Results and discussion

First, descriptive statistics are reported to describe the basic features of the data in this study. Second, the TOEICÒ scores are analyzed
using a two-way mixed design ANCOVA, in which learners’ additional learning hours were the covariate. Proficiency level grouping provides
three levels (advanced, intermediate and basic), and measurement occasion consists of both pretests and posttests. Considering the small
sample size, data from the basic level was excluded from ANCOVA analysis of TOEICÒ scores. Learners’ scores for CCTST and CCTDI, including
the results for each subscale, were analyzed using t-tests.

Fig. 5. Immediate, systematic feedback for listening comprehension questions showing feedback for both correct and incorrect responses.
 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 297

Fig. 6. The process for online discussions, uploading and management of files on the Moodle system, and individualized feedback.

3.1. Results

3.1.1. Critical thinking skills

Before t-tests were conducted, a Shapiro–Wilk test was used to evaluate the normality of CCTST data; a non-significant result (p ¼ .15)
indicates that the data were normally distributed. Means and standard deviations for CCTST scores are presented in Table 6, along with the
p values from t-tests. Significant increases in CTS were observed between the pretest and posttest (p ¼ .00). Moreover, the results indicate
a significant increase in overall CCTST performance as well as improvement on all of the subscales, including analysis, inference, evaluation,
induction, and deduction.
298 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

Fig. 6. (continued).

3.1.2. Critical thinking dispositions

Before t-tests were conducted, a Shapiro–Wilk test was used to evaluate the normality of CCTDI data; a non-significant result (p ¼ .46)
indicates that the data were normally distributed. Means and standard deviations for CCTDI scores, along with the p values from t-tests, are
presented in Table 7. A statistically significant increase in open-mindedness (p ¼ .04) was observed. The findings for the other subscales,
however, were not significant. Overall, very small differences were observed between the pretest and posttests for CTD.

3.1.3. English listening

Means and standard deviations for the three proficiency level groups are reported in Table 8 (see also Fig. 7). Excluding the basic group,
the overall performance for English listening proficiency was analyzed using a 2  2 mixed design ANCOVA. Before ANCOVA was conducted,
 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 299

Fig. 6. (continued).

a Shapiro–Wilk test was used to evaluate the normality of English listening scores; a non-significant result (p ¼ .16) indicates that the data
were normally distributed. Additionally, a test for homogeneity of variance was conducted, which revealed no significant effect, F ¼ 1.13,
p ¼ .34; that is, the data meet the requirement for homogeneity of variance. As Table 9 shows, by including learners’ additional listening
hours as a covariate, the results of the analysis of covariance (ANCOVA) indicate a significant difference between the listening pretest and
posttest, F ¼ 40.29, p ¼ .00, h2 ¼ .34. The interaction between occasion and group was also significant, F ¼ 10.18, p ¼ .00, h2 ¼ .12. As
described by Cohen (1988), h2  .01 indicates a small effect size; h2  .06 indicates a moderate effect size; and h2  14 represents a large
effect size. As such, the main effect and interaction effect were large. Although the covariate was not statistically significant (F ¼ .28, p ¼ .60),
it was helpful in adjusting for differences between the listening pretest and posttest.
Post hoc analysis demonstrates that the posttest mean score was significantly higher than the pretest mean score for the intermediate
group (t ¼ 7.05, p ¼ .00, d ¼ 1.16) and advanced group (t ¼ 2.78, p ¼ .01, d ¼ .40). As described by Cohen (1988), d ¼ .20 indicates a small effect
size; d ¼ .50 indicates a moderate effect size; and d ¼ .80 represents a large effect size. These results indicate that the instructional treatment
resulted in significant improvement for both intermediate and advanced learners’ English listening proficiency. Based on the significant
interaction effect, it appears that the intermediate group improved to a greater degree than the advanced group. However, this is unsur-
prising since members of the advanced group scored nearer to the maximum score of the test than intermediate group members. Thus, the
mean score of the advanced group improved by 17.24 points to 430.65 on the posttest, which approaches a maximum score of 495, while
intermediate group improved by 51.79 points to 360.74.

3.1.4. English speaking

The mean scores and standard deviations of the three level groups are reported in Table 8 (see also Fig. 8). Excluding the basic group,
overall performance in English speaking proficiency was analyzed using a 2  2 mixed design ANCOVA. Before ANCOVA was conducted,
a Shapiro–Wilk test was used to test the normality of English speaking scores; a non-significant result (p ¼ .33) indicate that the data were
normally distributed. Additionally, a test for homogeneity of variance was conducted, revealing no significant effect (F ¼ .91, p ¼ .44); that is,
the data meet the requirement for homogeneity of variance. As shown in Table 10, by including learners’ additional listening hours as
a covariate, the results of the analysis of covariance (ANCOVA) indicate a significant difference between the listening pretest and posttest,
F ¼ 48.54, p ¼ .00, h2 ¼ .41. The interaction between occasion and group was also significant, F ¼ 12.68, p ¼ .00, h2 ¼ .15. According to Cohen
(1988), these effect sizes can be described as large. Although the covariate was not statistically significant (F ¼ .13, p ¼ .72), it was helpful in
adjusting for differences between the speaking pretest and posttest.
Post hoc analysis demonstrates that the posttest mean scores were significantly higher than the pretest means scores for the inter-
mediate group (t ¼ 9.53, p ¼ .00, d ¼ 1.60) and the advanced group (t ¼ 2.35, p ¼ .03, d ¼ .53). According to Cohen (1988), these effect sizes

Table 6
Means (M), standard deviations (SD), and t-test significance for CCTST scores.

CCTST scale Pretest Posttest p d

M SD M SD
Analysis 4.36 1.33 4.89 1.10 .00* .43
Inference 6.49 1.89 7.11 1.67 .00* .35
Evaluation 6.37 2.14 7.96 1.88 .01* .80
Induction 6.82 2.18 7.73 1.69 .01* .47
Deduction 8.75 2.38 10.40 2.42 .00* .69
CCTST overall 17.23 3.94 19.96 3.37 .00* .74
*
p < .05.
300 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

Table 7
Means (M), standard deviations (SD), and t-test significance for CCTDI scores.

CCTDI scale Pretest Posttest p d

M SD M SD
Truth-seeking 30.19 5.36 31.01 4.99 .11 .16
Open-mindedness 37.15 3.94 38.18 4.44 .04* .25
Analyticity 41.84 5.35 41.93 5.86 .87 .02
Systematicity 36.74 5.55 36.94 6.36 .72 .03
CT self-confidence 43.46 7.02 42.24 7.16 .06 .17
Inquisitiveness 49.09 5.46 49.43 5.52 .52 .06
Maturity of judgment 40.29 6.23 41.25 5.82 .09 .16
CCTDI overall 278.75 25.00 280.96 26.63 .30 .09
*
p < .05.

would be considered large. These results indicate that the instructional treatment improved intermediate and advanced learners’ English
speaking proficiency. Based on the significant interaction effect, it appears that the intermediate group improved to a greater degree than
the advanced group. However, this is unsurprising since members of the advanced group scored nearer to the maximum score of the test
than intermediate group members. Thus, the mean score of the advanced group improved by 9.03 points to 158.65 on the posttest, which
approaches a maximum score of 200, while intermediate group improved by 27.34 points to 126.38.

3.2. Discussion

3.2.1. Improvement in critical thinking skills

The blended CT-integrated individualized English instruction had a positive impact on learners’ CTS. Improvement in each subscale of the
CTS, namely analysis, inference, evaluation, induction, and deduction was noted after the treatment. This may be due to the direct CTS
instruction, which coincides with the suggestions of previous research (e.g., Gamble & Yang, 2010; Marin & Halpern, 2011). Since direct and
explicit instruction and practice activities were provided for each of the above skills respectively, learners not only had a clear idea of what
these skills involved, but also were able to put them into practice. Moreover, according to Hummel and Holyoak (1997), thinking is structure
sensitive. Therefore, explicit instruction helps learners transfer knowledge across disciplines (Halpern, 1999). For example, one of the CT
skills for week eight was inductive reasoning in the context of an office. The instructor introduced the concept of inductive reasoning and
then learners worked face-to-face in groups to discuss the sample issue related to the office. After class, they had more opportunities for
practice with the assistance of the online learning package. In this way, learners had a chance to learn and understand the concept clearly
before analyzing and evaluating what they might encounter in the office.
In addition to the direct instruction for each individual CT skill, the present study also implemented structured synchronous online
discussions with Socratic questioning in order to enhance learners’ CTS; with positive results coinciding with those of Yang, Newby, and Bill
(2005). Teaching Socratic questioning through generic questioning prompts is effective in promoting learners’ CT (King, 1995). In the present
study, before learners began discussions in the synchronous online discussion activities, the instructor introduced the six categories of
Socratic questioning. Following direct instruction and discussion, learners worked in groups to discuss with their partners how to apply
Socratic questioning. After learners discussed their ideas in the synchronous online discussion boards, the instructor provided individu-
alized feedback on their performance. This direct and immediate feedback follows the suggestions of McLoughlin and Luca (2000), who
argue that teachers or tutors can scaffold individuals to develop CT by offering questions and direct feedback.
The positive effect of CT-integrated individualized English instruction on learners’ CTS acquisition further echoes the belief that language
and CT are inseparable (see Carr, 1988; Rath, Jonas, Rothstein, & Wassermann, 1986). That is to say, learners can be taught to think critically,
to analyze and make comparisons, and to question and evaluate through language learning.

3.2.2. Mixed results for critical thinking dispositions

Due to minimal changes in the CCTDI scores between the pretest and posttest, it is difficult to conclude that the intervention had any
effect upon critical thinking dispositions. We can note, however, that the mean scores suggest a relatively weak level of CTD for our
participants. According to Facione, Facione, and Giancarlo (1996), scores of less than 40 on any scale or less than 280 overall are indicative of
a deficiency in the disposition toward critical thinking. With a pretest overall mean score of 278.76 and posttest mean score of 280.96, we
might conclude that our participants’ relative lack of attitudes and dispositions toward critical thinking required a different approach to CTD
cultivation. Additionally, as several authors (e.g., Yang & Chou, 2008) have suggested, CTD cultivation requires a longer period of time. As

Table 8
Means (M) and standard deviation (SD) for overall listening and speaking performance, with significant values from post-hoc analysis.

Levels n Pretest Posttest p d

M SD M SD
Listening Basic 3 132.00 30.05 221.00 73.91 N.A. N.A.
Intermediate 43 308.95 45.91 360.74 42.97 .00* 1.16
Advanced 37 413.41 36.73 430.65 41.39 .01* .40
Speaking Basic 8 45.00 .00 70.63 31.56 N.A. N.A.
Intermediate 49 99.04 16.83 126.38 16.04 .00* 1.60
Advanced 26 149.62 13.92 158.65 19.62 .03* .53
*
p < .05.
N.A. ¼ not available.
 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 301

Fig. 7. Improvement in listening between the pretest and posttest for all groups.

dispositions are affective in nature, the types of practice and exposure to CT concepts that were effective in fostering CTS may have been
ineffectual in touching the deeper attitudes and feelings toward CT in general.
One subscale of the CCTDI did show a statistically significant improvement, although only a small increase in the raw score was observed.
Open-mindedness, refers to one’s tolerance for divergent viewpoints, seems to have been slightly enhanced by the CT-integrated English
instruction, although not to a level above the 40 point cut-off representing weakness in a particular CTD. Possible reasons may include the
relatively homogeneous nature of Taiwan’s education system and pedagogical traditions, and the fact that having new and innovative
activities (such as debates) allowed students to open their minds to new ideas and perspectives. Our intention in designing a virtual learning
environment was to provide learners with several authentic opportunities for sharing and discussing opinions, perspectives, and experi-
ences in interaction with their peers. These elements of expression and interaction with the viewpoints of others are central to developing
open-mindedness. Through engagement with different perspectives, some learners may have developed an open mind to multiple
perspectives. However, whether open-mindedness is more easily influenced than other elements of CTD remains unclear, and further
studies are needed before any conclusions can be drawn.
Overall CTD scores in the current study demonstrated no improvement, despite significant improvement in CTS. While Yang and Chou
(2008) concluded that the success of CTS instruction strengthens CTD, several factors may have limited learner cultivation of CTD. Most
significantly, the time allowed for CTD cultivation (in this case one semester) was far too short to expect a noteworthy improvement.
Secondly, while the instruction of CTS was expected to lead to improvements in CTD, the activities designed specifically for fostering CTD
may have been too few (i.e., primarily in the second half of the course), and too difficult to personalize (i.e., the examples provided for CTD
were in English, which is not the learners’ native tongue). However, the difficulty associated with the development of strategies to improve
CTD may suggest that influencing dispositions, which are by definition tendencies or beliefs, requires a more intensive, direct, and indi-
vidualized approach. Thus, while our study individualized CT-integrated English activities according to listening and speaking proficiency
scores, a similar approach may be necessary for taking into account learners’ CTD scores.

3.2.3. Improvements in listening and speaking

The results indicate that online CT-integrated individualized English instruction had a positive impact on learners’ English listening and
speaking. In relation to the research questions and procedure, as well as the related literature, possible reasons for improvements in
learners’ English proficiency are described as follows.
Grouping learners based on their pretest performances allowed learners to receive listening and speaking instruction, materials, and
activities corresponding to their abilities. The content of the articles, questions, and follow-up activities were all adapted to learners’ English
proficiency levels, following suggestions from Wang and Walberg (1985), Wang and Lindvall (1984) and Waxman, Wang, Anderson, and
Walberg (1985). Learners of different levels were thereby able to learn with adaptations for their English proficiency, decreasing cogni-
tive overload. Moreover, to give learners a richer and more varied educational experience, the research team also provided additional
learning resources for each group to use, a form of “enrichment”. By making use of these additional learning resources and participating in
further discussion online, learners became actively engaged and extended their knowledge, thinking, and views on issues (Reis, Eckert,
McCoach, Jacobs, & Coyne, 2008).

Table 9
Summary of group (2)  occasion (2) mixed design ANCOVA for listening scores (excluding basic group).

SV SS df MS F p Partial h2
Covariate 265.12 1 265.12 .28 .60 .00
Occasion (B) 38,799.18 1 38,799.18 40.29 .00* .34
Group  occasion (A  B) 9797.48 1 9797.48 10.18 .00* .12
Occasion  error (B  S/A) 74,144.84 77 962.92
*
p < .05.
302 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

Fig. 8. Improvement in speaking between the pretest and posttest for all groups.

Furthermore, individualized feedback enabled learners to avoid errors and acquire correct language patterns, resulting in a better
understanding of the content and more accurate responses in the future (Jaehnig & Miller, 2007). Two kinds of feedback were included in the
current study: feedback from instructors, teacher assistants, and peers and feedback directly from the online system. During a speaking
session, for example, the English instructor provided learners face-to-face comments by pointing out errors, which helped increase
accuracy. In addition, as with previous research which suggests that learning gains increase with the informative feedback provided (Hwa,
Fook, Atan, Majid, & Luan, 2007; Jaehnig & Miller, 2007), the online individualized system was programmed to provide multiple choice
review questions for each unit, which allowed learners to receive simultaneous feedback from the system itself. In addition to correct
answers, more information or an explanation about the correct answer was provided, for example, “Please try again! The speaker said
movie, not music. It’s a trap.”
In terms of Walberg’s (1975) individualized instructional categorized elements, we can also explain the significant improvement in oral
communication skills. For selection, learners’ needs were taken into account by providing materials and activities matched with their
learning background and needs through proficiency grouping, For enrichment, the virtual learning environment provided a variety of
materials that included audio-visual and digital content, allowed access to TA and instructor feedback, and fostered collaboration through
online discussions. Acceleration was achieved through the feedback mechanisms mentioned above, as well as though the self-paced nature
of online learning. This self-paced design allowed participants to repeat modules until they achieved mastery of skills, and accommodated
the busy schedules of undergraduate students by enabling participants to complete modules at their own convenience.

4. Limitations of the study and recommendations for future research

Although the results of our study highlight significant improvements in English and critical thinking resulting from a virtual learning
environment for individualized instruction, certain limitations and areas for future improvement and research should be mentioned. For
example, use of comparison groups, advancements in individualized instruction, and applications for commercial and large-scale use are
suggested as areas for future research and improvement.

4.1. The use of a comparison group

No comparison group was used in examining whether this online individualized instruction fostered greater improvement in English
proficiency and CT than an alternative method of teaching, such as traditional face-to-face instruction or a non-individualized online
learning course. It therefore may still be difficult to fully identify the effect of the online individualized instruction (Johnson & Christensen,
2008) on the learning outcomes.
Furthermore, the current study involved multiple types of interventions simultaneously (i.e., direct English instruction, direct CT
instruction, CT-integrated instruction, and online adaptive learning), making it difficult to draw conclusions regarding the influence of

Table 10
Summary of group (2)  occasion (2) mixed design ANCOVA for speaking scores (excluding basic group).

SV SS df MS F p Partial h2
Covariate 25.08 1 25.08 .13 .72 .00
Occasion (B) 9497.46 1 9497.46 48.54 .00* .41
Group  occasion (A  B) 2480.17 1 2480.17 12.68 .00* .15
Occasion  error (B  S/A) 13,696.68 70 195.67
*
p < .05.
 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 303

specific intervention types on specific outcome variables. Future studies using comparison groups assist in more clearly distinguishing the
contribution of each element of the instructional design.

4.2. Refining individualization or grouping strategies

Although the online individualized system used for this study had the capability of putting automated groupings of learners into
proficiency level groups, the TOEICÒ copyright limitations do not allow for the inclusion of test questions in an online format. Therefore, in
this study, the grouping was performed manually at the initial stage. Future study incorporating Item Response Theory (IRT) to develop an
entirely online proficiency level grouping algorithm should provide further confirmation of the utility of such an online individualized
learning system for incorporating CT training into English courses. Alternatives to grouping which provide full individualization of the
online course have also been proposed. Hsieh, Wang, Su, and Lee (2012) have recently developed and tested a personalized learning system
for online English learning using a form of IRT to recursively generate a reading proficiency level for learners and automatically generate
adaptive content. While such a system would be difficult for speaking proficiency, which requires human evaluation, listening compre-
hension test data could potentially be used to allow a greater degree of individualization for instruction in VLEs.

4.3. Individualized instruction for critical thinking

In this study, the CTS and CTD activities did not take full advantage of the online individualized design. That is to say, there were no
individualized instructional strategies or materials on the pretest specifically for learners with different levels of CTS or CTD. Moving toward
a more fully individualized design could allow for deeper CT fostering activities for high CT learners, while emphasizing the fundamentals of
CT and reasoning for lower CT learners. This could lead to a potential increase in CTD scores for learners when materials and activities are
tailored to their level (in our study, a relatively low average disposition).

4.4. Applications for commercial and large-scale use

While proficiency level grouping was one solution to meet the diverse learning needs of our learners, certain difficulties arise when
applying this procedure to smaller class sizes. Statistically, the number of students in the basic level was quite small, thus a comparison
among levels was not possible. Practically, having only three or eight students in one proficiency level makes it difficult for online discussion
grouping (assuming that a five-member group is preferred). Finally, the expense of time and resources on developing a virtual learning
environment for individualized instruction is better suited to larger-scale applications, such as for the online distribution of commercial
content. Alternatively, large-scale academic application of adaptive learning systems, such as CT-integrated freshman English classes for
a large university with several sections, may better justify the expense and time required in designing online adaptive learning systems.

5. Conclusions

Given the advent technology and Internet, using English and CT as communication tools and learning equipment are more important
than ever before. The study integrated CT into online individualized English listening and speaking instruction for learners at basic,
intermediate, and advanced levels grouped by their pretest TOEICÒ scores. The purposes of this study were to examine the effects of a CT-
integrated online individualized English listening and speaking course on: (a) CTS, (b) CTD, (c) English listening proficiency, and (d) English
speaking proficiency.
In terms of the first goal, we found that participants in the treatment showed significant improvement in CTS from pretest to posttest.
This is attributed to both the direct and explicit instruction of CTS concepts with accompanying practice activities and a core of CT-integrated
English listening and speaking activities which included online debates, synchronous discussions, and Socratic questioning. In terms of the
second goal, the empirical results of this study showed that learners’ CTD performance changed very little over the course of the treatment.
Among the seven CCTDI subscales, a significant improvement was only found for one subscale: open-mindedness. This disappointing result
can be partially explained by participants’ low initial levels of CTD, insufficient time and emphasis on CTD development, and the lack of an
individualized approach for CTD instruction. Finally, the empirical results of this study indicate that our designed VLE assisted learners in
improving in both English listening and speaking. We attribute this to the design which addressed learners’ need for authentic language
experiences, student-centered learning, and attention to learner needs. This was accomplished through individualized instruction using
proficiency grouping, design of materials and activities for each level, and use of individualized feedback.
The findings have implications both for instructional designers and instructors. Instructional designers can successfully develop CT-
integrated models and individualized English course materials based upon the model of individualized instruction presented in this
paper. In order to align with the needs, backgrounds, and interests of learners, VLEs can be used to achieve proficiency grouping, access to
authentic materials, and opportunities for collaborative and engaging student-centered activities. For instructors of English, particularly in
English as a Second Language (ESL) or English as a Foreign Language (EFL) contents, an emphasis on CT-integrated instruction can benefit
learners of all levels. As two prominent 21st century skills, CT and English can be fostered simultaneously, by paying attention to the
learners’ needs for authentic language interactions, ability-matched materials, and positive, helpful feedback.

Acknowledgements

Funding for this research was provided by the National Science Council of Taiwan, under grants NSC 96-2520-S-006-003-MY3 and NSC
99-2628-S-006-001-MY3.
304 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305

References

AbuSeileek, A. F. (2007). Cooperative vs. individual learning of oral skills in a CALL environment. Computer Assisted Language Learning, 20(5), 493–514.
Alagozlu, N. (2007). Critical thinking and voice in EFL writing. Asian EFL Journal, 9(3), 118–136.
Al-Ajlan, A., & Zedan, H. (2008). Why Moodle? In Proceedings of the 12th IEEE international workshop on future trends of distributed computing systems Los Alamitos, CA: IEEE
Computer Society
An, Y. J. (2006). Collaborative problem-based learning in online environments. Indiana University. Unpublished doctoral dissertation.
Barak, M., Ben-Chaim, D., & Zoller, U. (2007). Purposely teaching for the promotion of higher order thinking skills: a case of critical thinking. Research in Science Education, 37,
353–369.
Bean, J. C. (2011). Engaging ideas: The professor’s guide to integrating writing, critical thinking, and active learning in the classroom (2nd ed.). San Francisco, CA: Jossey-Bass
Publishers.
Benesch, S. (1999). Thinking critically, thinking dialogically. TESOL Quarterly, 33(3), 573–580.
Birjandi, P., & Bagherkazemi, M. (2010). The relationship between Iranian EFL teachers’ critical thinking ability and their professional success. English Language Teaching, 3(2),
135–145.
Brandl, K. (2005). Are you ready to “Moodle”? Language Learning & Technology, 9(2), 16–23.
Carr, K. S. (1988). How can we teach critical thinking? Childhood Education, 65(2), 69–73.
Chapple, L., & Curtis, A. (2000). Content-based instruction in Hong Kong: student responses to film. System, 28, 419–433.
Chau, J. P. C., Chang, A. M., Lee, I. F. K., Ip, W. Y., Lee, D. T. F., & Wootton, Y. (2001). Effects of using videotaped vignettes on enhancing students’ critical thinking ability in
a baccalaureate nursing programme. Journal of Advanced Nursing, 36(1), 112–119.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: L. Erlbaum Associates.
Corich, S. (2005). Is it time to Moodle? In S. Mann, & T. Clear (Eds.), The 18th annual NACCQ conference (pp. 155–158). Tauranga, New Zealand: NACCQ.
Crismore, A. (2000). Helping ESL and EFL university students read critically: A 2000’s challengeIn ERIC Document Reproduction Service No. ED450592.
Dantas-Whitney, M. (2002). Critical reflection in the second-language classroom through audiotaped journals. System, 30, 543–555.
Davidson, B., & Dunham, R. (1997). Assessing EFL student progress in critical thinking with the Ennis-Weir Critical Thinking Essay Test. JALT Journal, 19(1), 43–57.
Dougiamas, M., & Taylor, P. C. (2003). Moodle: Using learning communities to create an open source course management system. In Proceedings of ED-MEDIA 2003, Retrieved
from. http://dougiamas.com/writing/edmedia2003/.
Eshel, Y., & Kohavi, R. (2003). Perceived classroom control, self-regulated learning strategies, and academic achievement. Educational Psychology, 23, 249–260.
Facione, P. A. (1990a). The California critical thinking skills test (CCTST): Form A. Millbrae, CA: California Academic Press.
Facione, P. A. (1990b). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction. Millbrae, CA: The California Academic Press.
Facione, P. A. (1992). The California critical thinking skills test (CCTST): Form B. Millbrae, CA: California Academic Press.
Facione, P. A. (2007). Critical thinking: What it is and why it counts. Millbrae, CA: California Academic Press.
Facione, P. A., Facione, C. F., & Giancarlo, C. A. (1996). The California critical thinking disposition inventory test manual. Millbrae, CA: California Academic Press.
Facione, P. A., Sanchez, C. A., Facione, N. C., & Gainen, J. (1995). The disposition toward critical thinking. Journal of General Education, 44(1), 1–25.
Fahim, M., Bagherkazemi, M., & Alemi, M. (2010). The relationship between test takers’ critical thinking ability and their performance on the reading section of TOEFLÒ.
Journal of Language Teaching and Research, 1(6), 830–837.
Ferreira, A., & Atkinson, J. (2009). Designing a feedback component of an intelligent tutoring system for foreign language. Knowledge-Based Systems, 22(7), 496–501.
Fisher, A. (2001). Critical thinking. Cambridge, UK: Cambridge University Press.
Florez, M. C. (2000). Critical literacy in the adult ESL classroomIn ERIC Document Reproduction Service No. ED438740.
Gamble, J. H., & Yang, Y.-T. C. (2010). Fostering critical thinking and information literacy in English as a Second Language classes. In One voice international collection of
scholarly works (pp. 117–128). Glenside, PA: One Voice Institute of Elemental Ethics and Education.
Gamoran, A., Nystrand, M., Berends, M., & LePore, P. C. (1995). An organizational analysis of the effects of ability grouping. American Educational Research Journal, 32(4),
687–715.
Ghaemi, H., & Taherian, R. (2011). The role of critical thinking in EFL teachers’ teaching success. Modern Journal of Applied Linguistics, 3(1), 8–22.
Golding, C. (2011). Educating for critical thinking: thought-encouraging questions in a community of inquiry. Higher Education Research & Development, 30(3), 357–370.
Hadley, A. O. (1993). Teaching language in context. Boston, MA: Heinle & Heinle.
Halpern, D. F. (1997). Critical thinking across the curriculum. Mahwah, NJ: Erlbaum.
Halpern, D. F. (1999). Teaching for critical thinking: helping college students develop the skills and dispositions of a critical thinker. New Direction for Teaching and Learning,
80, 69–74.
Halx, M. D., & Reybold, L. E. (2005). A pedagogy of force: faculty perspectives of critical thinking capacity in undergraduate students. Journal of General Education, 54(4),
293–315.
Hölbl, M., & Welzer, T. (2010). Students’ feedback and communication habits using Moodle. Electronics and Electrical Engineering, 6(102), 63–66.
Hsieh, T.-C., Wang, T.-I., Su, C.-Y., & Lee, M.-C. (2012). A fuzzy logic-based personalized learning system for supporting adaptive English learning. Educational Technology &
Society, 15(1), 273–288.
Hsu, H. Y., Wang, S. K., & Comac, L. (2008). Using audioblogs to assist English-language learning: an investigation into student perception. Computer Assisted Language
Learning, 21(2), 181–198.
Hummel, J. E., & Holyoak, K. J. (1997). Distributed representations of structure: a theory of analogical access and mapping. Psychological Review, 104(3), 427–466.
Hwa, Y. L., Fook, F. S., Atan, H., Majid, O., & Luan, W. S. (2007). A study on different types of feedback in a language multimedia courseware. International Journal of Instructional
Technology & Distance Education, 4(4), 23–34.
Internet World Stats. (2011). Internet usage statistics: The big picture. Retrieved from. http://www.internetworldstats.com/stats.htm.
Ireson, J., & Hallam, S. (2001). Ability grouping in education. London: Paul Chapman Publishing.
Jaehnig, W., & Miller, M. L. (2007). Feedback types in programmed instruction: a systematic review. The Psychological Record, 57(4), 219–232.
Johnson, R. B., & Christensen, L. B. (2008). Educational research: Quantitative, qualitative, and mixed approaches. Thousand Oaks, CA: Sage Publications.
Karagiannidis, C., Sampson, D., & Cardinali, F. (2001). Integrating adaptive educational content into different courses and curricula. Educational Technology & Society, 4(3),
37–44.
Kennedy, M., Fisher, M. B., & Ennis, R. H. (1991). Critical thinking: literature review and needed research. In L. Idol, & B. Fly Jones (Eds.), Educational values and cognitive
instruction: Implications for reform (pp. 11–40). Hillsdale, NJ: Erlbaum.
Khan, B. H. (1997). Web-based instruction (WBI): what is it and why is it? In B. H. Khan (Ed.), Web-based instruction (pp. 5–18). Englewood Cliffs, NJ: Educational Technology.
Kim, H. K. (2008). Beyond motivation: ESL/EFL teachers’ perceptions of the role of computers. CALICO Journal, 25(2), 241–259.
King, A. (1995). Designing the instructional process to enhance critical thinking across the curriculum: inquiring minds really do want to know: using questioning to teach
critical thinking. Teaching of Psychology, 22(1), 13–17.
Kovalik, D. L., & Kovalik, L. M. (2007). Language simulations: the blending space for writing and critical thinking. Simulation and Gaming, 38(3), 310–322.
Kurfiss, J. (1988). Critical thinking: Theory, research, practice, and possibilitiesIn ASHE-ERIC Higher Education Report No. 2. Washington, DC: Association for the Study of Higher
Education.
Larsen-Freeman, D. (2000). Techniques and principles in language teaching. New York, NY: Oxford University Press.
Lund, A. (2008). Wikis: a collective approach to language production. ReCALL, 20(1), 35–54.
McBride, R. E., Xiang, P., Wittenburg, D., & Shen, J. (2002). An analysis of preservice teachers’ dispositions toward critical thinking: a cross-cultural perspective. Asia-Pacific
Journal of Teacher Education, 30(2), 131–140.
McLoughlin, C., & Luca, J. (2000). Cognitive engagement and higher order thinking through computer conferencing: We know why but do we know how? In Conference
proceedings of the 9th Annual Teaching Learning Forum. Perth, Australia: Curtin University of Technology.
Marin, L., & Halpern, D. F. (2011). Pedagogy for developing critical thinking in adolescents: explicit instruction produces greatest gains. Thinking Skills and Creativity, 6, 1–13.
Meurant, R. C. (2010). How computer-based internet-hosted learning management systems such as Moodle can help develop L2 digital literacy. International Journal of
Multimedia and Ubiquitous Engineering, 5(2), 1–7.
Mok, K.-H. (2010). The search for new governance of higher education in Asia. Hampshire, UK: Palgrave Macmillan.
 Y.-T.C. Yang et al. / Computers & Education 63 (2013) 285–305 305

Nagappan, R. (2001). Language teaching and the enhancement of higher-order thinking skills. In W. A. Renandya, & N. R. Sunga (Eds.), Anthology series 42 – Language
curriculum and instruction in multicultural societies. Singapore: SEAMEO Regional Language Center.
Novak, J. D. (2010). Learning, creating, and using knowledge: Concept maps as facilitative tools in schools and corporations (2nd ed.). New York, NY: Routledge.
Nunan, D. (2003). The impact of English as a global language on educational policies and practices in the Asia-Pacific region. TESOL Quarterly, 37(4), 589–613.
Pally, M. (1997). Critical thinking in ESL: an argument for sustained content. Journal of Second Language Writing, 6(3), 293–311.
Partnership for 21st Century Skills. (2011). Learning for the 21st century: A report and mile guide for 21st century skills. Retrieved from. http://www.p21.org/storage/documents/
P21CommonCoreToolkit.pdf.
Paul, R. (1995). Socratic questioning and role-playing. Santa Rosa, CA: Foundation for Critical Thinking.
Paul, R., & Elder, L. (2007). Critical thinking: the art of Socratic questioning. Journal of Developmental Education, 31(1), 36–37.
Peterson, E. (2009). Teaching to think: applying the Socratic method outside the law school setting. Journal of College Teaching & Learning, 6(5), 83–87.
Phan, H. P. (2009). Exploring students’ reflective thinking practice, deep processing strategies, effort, and achievement goal orientations. Educational Psychology, 29(3), 297–313.
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the Motivated Strategies for Learning QuestionnaireIn Tech. Rep. No. 91-B-004. Ann
Arbor, MI: University of Michigan.
Rath, L. E., Jonas, A., Rothstein, A., & Wassermann, S. (1986). Teaching for thinking: Theory, strategies & activities for the classroom. New York, NY: Teachers College Press.
Recker, M., Ram, A., Shikano, T., Li, G., & Stasko, J. (1995). Cognitive media types for multimedia information access. Journal of Educational Multimedia and Hypermedia, 4(2),
183–210.
Reis, S. M., Eckert, R. D., McCoach, D. B., Jacobs, J. K., & Coyne, M. (2008). Using enrichment reading practices to increase reading fluency, comprehension, and attitudes. Journal
of Educational Research, 101(5), 299–314.
Richards, J. C., & Rogers, T. S. (2003). Approaches and methods in language teaching (2nd ed.). Cambridge, UK: Cambridge University Press.
Rodriguez, C. M. (2009). The impact of academic self-concept, expectations and the choice of learning strategy on academic achievement: the case of business learners. Higher
Education Research and Development, 28(5), 523–539.
Rosenberg, M. J. (2001). E-learning: Strategies for delivering knowledge in the digital age. New York, NY: McGraw Hill.
Shaaban, K. (2005). A proposed framework for incorporating moral education into the ESL/EFL classroom. Language, Culture and Curriculum, 18(2), 201–217.
Shang, Y., Shi, H. C., & Chen, S. S. (2001). An intelligent distributed environment for active learning. ACM Journal of Educational Resources in Computing, 1(2), 1–17.
Shute, V. J., & Towle, B. (2003). Adaptive e-learning. Educational Psychologist, 38(2), 105–114.
Tishman, S., & Andrade, A. (1999). Thinking dispositions: A review of current theories, practices, and issues. Retrieved from. http://learnweb.harvard.edu/alps/thinking/docs/
Dispositions.pdf.
Tishman, S., Jay, E., & Perkins, D. (1993). Teaching thinking disposition: from transmission to enculturation. Theory into Practice, 32, 147–153.
Walberg, H. J. (1975). Psychological theories of educational individualization. In H. Talmage (Ed.), System of individualized education (pp. 15–18). Berkeley, C.A.: McCutchan.
Walberg, H. J. (1984). Improving the productivity of American schools. Educational Leadership, 41, 19–27.
Wang, Y. H. (2009). Incorporating critical thinking skills into an English conversation program. European Journal of Social Sciences, 11(1), 51–60.
Wang, M. C., & Lindvall, C. M. (1984). Individual differences and school learning environments. Review of Research in Education, 11, 161–225.
Wang, M. C., & Walberg, H. J. (1983). Adaptive instruction and classroom time. American Educational Research Journal, 20(4), 601–626.
Wang, M. C., & Walberg, H. J. (1985). Adapting instruction to individual differences. Berkeley, CA: McCutchan.
Waxman, H. C., Wang, M. C., Anderson, K. A., & Walberg, H. J. (1985). Adaptive education and student outcomes: a quantitative synthesis. Journal of Educational Research, 78,
228–236.
Wen, J. R., & Shih, W. L. (2008). Exploring the information literacy competence standards for elementary and high school teachers. Computers & Education, 50(3), 787–806.
Wu, W. S. (2008). The application of Moodle on an EFL collegiate writing environment. Journal of Education and Foreign Languages and Literature, 7, 45–56.
Wu, W.-C. V. (2009). Criteria for establishing an authentic EFL learning environment in Taiwan. Asian EFL Journal, 11(3), 156–189.
Yang, C. X. (2002). Establish the English learning environment. National Policy Forum (Taiwan), 2(6), 159.
Yang, Y.-T. C. (2008). A catalyst for teaching critical thinking in a large university class in Taiwan: asynchronous online discussions with the facilitation of teaching assistants.
Educational Technology Research and Development, 56(3), 241–264.
Yang, Y.-T. C., & Chou, H.-A. (2008). Beyond critical thinking skills: Investigating the relationship between critical thinking skills and dispositions through different online
instructional strategies. British Journal of Educational Technology, 39(4), 666–684.
Yang, Y.-T. C., Gamble, J., & Tang, S.-Y. S. (2012). Voice over instant messaging as a tool for enhancing the oral proficiency and motivation of English-as-a-Foreign-Language
learners. British Journal of Educational Technology, 43(3), 448–464.
Yang, Y.-T. C., Newby, T. J., & Bill, R. L. (2005). Using Socratic questioning to promote critical thinking skills through asynchronous discussion forums in distance learning
environments. American Journal of Distance Education, 19(3), 163–181.
Yeh, Y.-C. (2004). Nurturing reflective teaching during critical thinking instruction in computer stimulation program. Computers & Education, 42(2), 181–194.
Yeh, Y.-C. (2012). A co-creation blended KM model for cultivating critical-thinking skills. Computers & Education, 59(4), 1317–1327.
Yu, Y., & Wang, B. (2009). A study of language learning strategy use in the context of EFL curriculum and pedagogy reform in China. Asia Pacific Journal of Education, 29(4),
457–468.
Şendag, S., & Odabaşɪ, H. F. (2009). Effects of an online problem based learning course on content knowledge acquisition and critical thinking skills. Computers & Education, 53,
132–141.