Available online at www.sciencedirect.

com

ScienceDirect
Cognitive Systems Research 58 (2019) 387–397
www.elsevier.com/locate/cogsys

Intelligent technologies to optimize performance: Augmenting

cognitive capacity and supporting self-regulation of critical
thinking skills in decision-making

Anastasia Kitsantas a,⇑, Amy L. Baylor b,1, Suzanne E. Hiller c

a
George Mason University, US
b
National Science Foundation, US
c
Wingate University, US

Received 30 June 2019; accepted 4 September 2019

Available online 6 September 2019

Abstract

The purpose of the present paper is to (a) discuss how intelligent systems can augment cognitive capacity, and through self-regulation,
assist learners to engage in critical thinking and (b) provide an example that highlights the information-rich context of the work envi-
ronment and the role of self-regulation in working in partnership with technology to achieve peak performance. Self-regulation from a
social cognitive perspective is conceptualized as a fluid, cyclical process whereby learners use externally provided or self-generated feed-
back to evaluate and adjust their learning strategies. With advances in technology, this paper attempts to illustrate how workers in jobs
of the future will be supported in new ways to analyze data, make interpretations and draw inferences, evaluate situations, and make
decisions within a work context before, during, and post-work. Educational implications will be discussed.
Ó 2019 Elsevier B.V. All rights reserved.

1. Intelligent technologies to optimize performance: for innovative career trajectories, educational initiatives
Augmenting cognitive capacity and supporting self- emphasize the development of self-sufficient students with
regulation of critical thinking skills in decision-making heightened levels of self-regulation who can think critically.
Critical thinking skills enable learners to navigate new
Ever-changing economic, environmental, and societal areas and make effective decisions. Most importantly, crit-
needs spur a labor force tailor-made to adjust to the goals ical thinking is essential for reflective processes during
of an interactive, international community that incorpo- problem solving (Lau & Chan, 2018) and is a key factor
rates new, technologically-advanced work environments. in optimizing performance across diverse areas of
In an effort to encourage the world’s youth to prepare functioning.
Research (Phan, 2010) has shown that self-regulation
can serve as a vehicle for the development of critical and
⇑ Corresponding author. analytic thinking skills and that developing student self-
E-mail address: [email protected] (A. Kitsantas). regulatory skills results in an increase of critical thinking.
1
This material is based upon work supported by (while serving at) the Thus, it seems that the theoretical orientations of self-
National Science Foundation. Any opinion, findings, and conclusions or
regulated learning and critical thinking may be interrelated
recommendations expressed in this material are those of the author(s) and
do not necessarily reflect the views of the National Science Foundation. in that self-regulatory processes may cultivate critical

https://doi.org/10.1016/j.cogsys.2019.09.003
1389-0417/Ó 2019 Elsevier B.V. All rights reserved.
388 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397

thinking, while at the same time, critical thinking may facil- but rather as a fluid, cyclical process where students use
itate the development of self-regulatory processes. externally provided or self-generated feedback to evaluate
Advances in Artificial Intelligence (AI), machine learning, and adjust their learning strategies. Known as the Cyclical,
and analytics provide new ways for augmenting learner Self-Regulatory Feedback Loop Model (see Fig. 1), it con-
capacity to self-regulate and think critically via intelligent sists of three phases.
systems. Thus, in partnership with technology, these sys- The first phase, which is the forethought phase, includes
tems can augment or cognitively extend one’s capacity to two subprocesses that precede efforts to learn, including (a)
think critically and metacognitively. As such, these modern task analysis and (b) self-motivational beliefs. Task analy-
human-technology partnerships present new ways to opti- sis involves strategy planning for an upcoming task and is
mize performance with intelligent support. Individuals further subdivided into two classes: goal setting and strate-
develop self-regulatory competencies to think critically gic planning. Self-motivational beliefs are a critical compo-
when they have access to social supports such as embodied nent of the forethought phase and encompass goal
virtual agents or technology-based scaffolds. The goal of orientations, self-efficacy, interest/task value, and outcome
these systems could be to reduce cognitive load and aug- expectations. These self-motivational strategies are critical
ment individuals’ self-regulation to engage in critical think- for the third phase of this cycle, self-reflection. Learners
ing and motivate learners to focus on higher order with strong self-motivational beliefs typically demonstrate
thinking. In this writing, we examine these dynamics from the ability to reflect on strategies at the onset of failures,
the perspective of professional tasks required for individu- thereby demonstrating more resiliency in reattempting a
als to optimize their work and reach peak performance. In task.
fact, by compartmentalizing cognitive demands, learners The second phase, the performance phase, incorporates
on a job site (e.g., a restaurant server) can use externally the subprocesses of (a) self-control and (b) self-
generated data to improve performance and focus on dif- observation. These subprocesses reflect learning efforts
ferent skill sets. Intelligent technology-based systems have heavily rooted in metacognition (e.g., strategy use and
the capacity to bridge both required communication skills self-monitoring). During self-control, the learner engages
and content knowledge. A compelling aspect of intelligent in behaviors which enable them to perform a specific task,
systems is that as a learning tool it is possible to identify including self-instruction, self-imagery, attention focusing,
subtleties in job performance and then help an individual and task strategies. The self-observation subprocess is com-
self-reflect and modify behavior. prised of metacognitive monitoring and self-recording.
In the present paper, we (a) define self-regulated learn- During self-observation learners are aware of their pro-
ing and critical thinking skills; (b) outline overlapping ele- gress, record their progress, and make real time adjust-
ments between self-regulatory processes and critical ments to perfect their skills.
thinking; (c) discuss how intelligent systems can augment The third phase, self-reflection, includes mechanisms
cognitive capacity, and through self-regulation, assist occurring after learning or performance, such as self-
learners to engage in critical thinking; (d) provide a futur- evaluation and self-reactions (Zimmerman, 2000, 2013).
istic example of the role of technology to optimize perfor- The two subprocesses in the self-reflection phase include
mance, support self-regulated learning, and promote (a) self-judgment and (b) self-reaction. Self-judgment
critical thinking skills in a specific work context; and (e) includes the classes of self-evaluation and causal attribu-
discuss future directions for research to optimize perfor- tion whereas self-reaction is reflective of self-satisfaction/
mance across diverse areas of human functioning. affect adaptive/defensive inferences. The elements within
the self-reflection phase correspond most closely with crit-
2. Defining self-regulated learning and critical thinking skills ical thinking skills as individuals must be able to reflect
in relation to optimal performance accurately on performance and assess lack of success in
relationship to strategy selection rather than personal attri-
There are several models of self-regulated learning that butes (Cleary & Labuhn, 2013).
have been developed to understand how learners proac- These phases, forethought, performance, and self-
tively master their own learning (Schunk & Zimmerman, judgment, are interdependent during a dynamic exchange
1994). Across these models, self-regulated learners are indi- in which the learner attempts to become proficient at a task
viduals who approach tasks purposefully and strategically, or skill through a coordination of planning, performance,
set goals, select strategies, actively employ these strategies and reflection. For example, changes in the forethought
when engaged in the learning, and consistently evaluate processes influence behavior during the performance phase,
and reflect on the quality of their strategic efforts to learn which in turn, steers self-reflection phase processes. A self-
and perform. One of the most influential theoretical models regulatory cycle is completed when self-reflection processes
of self-regulated learning is a three-phase cyclical feedback influence subsequent forethought beliefs and behaviors
model (Zimmerman, 2000) which emphasizes the role of during future learning efforts. Research findings in aca-
metacognitive, motivational, and behavioral processes in demic and sport settings show that self-regulation is consis-
learning acquisition. According to this model, self- tently associated with high levels of performance (Kitsantas
regulation is not conceptualized as a stable, fixed ability, & Zimmerman, 2006, Zimmerman & Kitsantas 2014).
 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397 389

Fig. 1. Zimmerman’s Cyclical Self-regulatory Feedback Loop Model to optimize performance. Adapted from Handbook of Self-regulation (p.17), edited
by M. Boekaerts, P. R. Pintrich, & M. Zeidner, 2000, San Diego: Academic Press.

This model has implications for developing expertise and (1933) definition of reflective thinking as ‘‘active, persis-
has viable applications for the study of critical thinking tent, and careful consideration of any belief or supposed
skills to optimize performance in any setting. form of knowledge in the light of the grounds that support
Critical thinking skills are not mutually exclusive to self- it and further conclusions to which it tends constitutes
regulation but rather are essential for preparing the learn- reflective thought” (p. 118). From a scientific perspective,
ers for the next course of action in an effort to refine their critical thinking and evaluation steer interpretations which
skills to advanced levels. Those individuals who do not are central to forming scientific explanations based on evi-
have strong critical thinking skills may not have the ability dence (Lombardi, Bickel, Bailey, & Burrell, 2016). Thus,
to reflect on the effectiveness of strategies accurately. despite differences in definitions, theorists seem to agree
According to the Merriam-Webster Dictionary, the word that critical thinking involves self-regulatory judgments,
critical derives from the Greek ‘‘Kritikos,” which means thinking about how we think, and engaging in reflective
discerning judgment while the word analytic is also derived skepticism to judge the rationality of thinking. Critical
from the Greek ‘‘Analytikós,” which means proceeding by and analytic learners use a variety of forms of reasoning,
analysis. Conceptual definitions in the literature differ to engage in analysis, evaluation and inference, and make
some degree but all share these dictionary terms and some judgments (Partnership for 21st Century Skills, 2009;
key processes. For example, experts in Facione (1990) Del- Sternberg, Roediger, & Halpern, 2007).
phi study (American Philosophical Association, 1990) Critical thinking stems from contextually specific knowl-
defined critical thinking as ‘‘the process of purposeful, edge and application, the ability to analyze information,
self-regulatory judgment, which results in interpretation, decision making, and reflection. For example, in a pilot
analysis, evaluation and inference . . .” (p. 23). Nosich study of first year medical students studying physiology,
(2012) stated, ‘‘Critical thinking is different from just think- researchers concluded that students with limited frame-
ing. It is metacognitive—it involves thinking about your works of understanding of physiology were not able to
thinking” (p. 3). Others (Brookfield, 2012; Grushka, apply critical thinking skills effectively (Nguyen et al.,
McLeod, & Reynolds, 2005; Pedro, 2005) refer to critical 2017). Moreover, critical thinking is heavily dependent on
thinking as critical reflection primarily based on Dewey’s knowledge acquisition. In this paper, we focus on learners
390 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397

who have the acquired knowledge/expertise to perform a feedback, students’ motivational beliefs and social interac-
task and on how to optimize performance in the desired tions are included in self-regulated accounts of learning.
area. Various critical and analytic theorists differ to some degree
in their choice of key processes. However, there is signifi-
3. Examining the conceptual overlap and distinctiveness of cant overlap between processes of these constructs and def-
self-regulated learning and of critical and analytic thinking initions have become somewhat diluted. For example,
Facione’s definition of critical thinking (1990) includes pur-
An interesting question is what is the relationship poseful self-regulatory judgments that result in interpreta-
between critical and analytic thinking and self-regulatory tion, analysis, evaluation, and inference. These processes
processes? As described in the previous section, conceptual are similar to the classes of task analysis (forethought
frameworks outlining critical thinking skills are often asso- phase), self-judgments, self-evaluation, and self-reactions
ciated with the work of Facione and include interpretation, (self-reflection phase) in Zimmerman (2000) model of
analysis, evaluation, inference, and explanation. In an self-regulated learning.
extension of his work, self-regulation appears as another Learners engaged in self-judgment subprocesses focus
distinguishing critical thinking skill. In this type of frame- on both self-evaluation and causal attributions. Students
work, self-regulation reflects one of six critical thinking assess how well attempted performances corresponded with
skills while some research studies focus on reflection of crit- task analysis during the forethought phase to prepare for
ical thinking skills apart from self-reflection processes. It is the upcoming task (Zimmerman, 2013). In this case, the
a misnomer to consider critical thinking skills and self- interplay between self-judgments and causal attributions
regulatory processes as distinct, mutually exclusive con- influences how well a student reflects on strategy selection
structs. In addition, from a social cognitive framework, and performance. Effective self-regulatory processes at this
the role of self-regulation extends beyond one type of crit- phase result in the reshaping of strategies (Labuhn,
ical thinking skill. Rather, critical thinking skills most clo- Zimmerman, & Haselhorn, 2010). Students with weak
sely align with all three phases of the Cyclical, Self- self-motivational beliefs are more likely to reflect on factors
regulatory Feedback Loop Model as shown in Fig. 2. out of their control such as teacher effectiveness, genetic
Based on the conceptual definitions reviewed above, factors, or circumstance rather than focusing on the tar-
critical and analytic thinking centers on the truth or falsity geted strategies (Cleary & Labuhn, 2013; Zimmerman,
of a claim or conclusion rather than whether a skill such as 2008). In this dynamic, self-reactions closely link with
reading, writing, or a sport is being learned. Critical and motivation. Self-satisfaction in the outcome of a learning
analytic thinking also focuses on knowledge in terms of experience leads to heightened interest and motivation
logical processes whereas self-regulated learning views (Bol, Hacker, Walk, & Nunnery, 2012). Learners with
knowledge acquisition as a transaction between a learner adaptive inferences are more resilient and willing to reat-
and his or her personal feedback. In addition to cognitive tempt modified strategies whereas individuals with defen-
sive inferences may avoid challenging activities in the
future (Zimmerman, 2000).
Other researchers (Brookfield, 2012; Nosich, 2012)
noted that critical thinking is metacognitive and requires
the learner to self-monitor and reflect on their quality of
thinking which are similar to self-monitoring and evalua-
tion in the social cognitive perspective of self-regulation.
Although part of the relationship between critical and ana-
lytic thinking and self-regulatory processes may be
explained in that both theoretical orientations rely on pro-
cesses associated with metacognition and reflection, the
cyclical feedback analysis of self-regulation includes addi-
tional processes: goal setting, strategy use, social processes
such as help seeking, modeling, and cooperative learning as
well as motivational beliefs.
In summary, the current age of information demands
the acquisition of logical reasoning skills. The foundation
of professional practice across disciplines, where the appli-
cation of critically assessing whether or not thought pro-
cesses and assumptions are valid and sound, allows
individuals to make accurate decisions. However, drawing
correct logical inferences and making decisions is only part
Fig. 2. Critical thinking skills as a manifestation of the three-phase model of the cyclical model of self-regulation for mastery of a
of self-regulatory functioning. complex skill that requires repeated self-guided practice
 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397 391

and repetition. In fact, Phan (2010) argued that critical implemented as human-like social interfaces and can effec-
thinking can be thought of as one of many cognitive strate- tively serve as social models to positively impact beliefs
gies used by self-regulated learners. Taking this view into such as attitudes and self-efficacy (Baylor, 2011). The sys-
consideration, below we argue how intelligent tem then can explicitly simulate a social experience to
technology-based systems can augment one’s performance model desired self-motivational beliefs and behaviors.
by facilitating self-regulation and critical thinking. For example, an anthropomorphic agent can be designed
to appear visually similar to what the student aspires to
4. Augmenting cognitive capacity through intelligent systems be and then can simulate a social relationship and model
to support self-regulation and critical thinking during desired behaviors. In one set of studies, the goal was to
performance develop young women’s self-efficacy and interest in engi-
neering with an embodied virtual agent that was designed
Traditionally, in the support of learning and perfor- to be similar to what the student aspired to be (appearing
mance, intelligent systems serve as adaptive environments as young, cool, and attractive). Results indicated that this
that can provide personalized support (Durlach & agent as a social model positively impacted student beliefs,
Lesgold, 2012). Rather than envisioning these systems as stereotypes and self-efficacy in engineering and, in turn,
a one-way didactic relationship or as static scaffolds, of also positively impacted math performance (Plant,
interest is to view systems in their potential to augment Baylor, Doerr, & Rosenberg-Kima, 2009; Rosenberg-
capacity and ultimately provide figurative ‘‘superpowers” Kima, Baylor, Plant, & Doerr, 2008). Further, it may not
in the capacity to self-regulate and think critically. In fact, even be necessary for the underlying system to be particu-
it was Engelbart (1962) who first asserted the potential of larly intelligent, as the appearance of the agent alone (when
computer technologies to enhance human capabilities, designed correctly) can have significant impact as a social
where the intelligent system is not designed to replace model (Baylor, 2011).
human thinking but rather to augment human capacity. Moving to the performance phase, an intelligent system
In this way, advances in AI, machine learning, and analyt- can play a significant and important role. Here, the system
ics provide an opportunity to re-conceptualize the role of can augment capacity by providing ‘‘super-senses” for the
intelligent technologies to support better self-regulation performer, allowing him/her to detect areas where she
and engagement in critical thinking. may need to adjust behaviors to focus attention and stay
This section will discuss examples of how to operational- on task. In Borge, Ong, & Penstein Rosé, 2018’s model
ize subprocesses in Zimmerman (2000) Cyclical, Self- of ‘‘self-regulation in the wild” the authors proposed that
regulatory Feedback Loop Model through augmenting facilitating attention is a key gateway for self-regulatory
self-regulation with the support of intelligent systems. Of processes to occur, ultimately leading to awareness, recog-
note is that in this conceptualization the learner works in nition of the problematic state, problem appraisal, apply-
partnership with the technology. Given that this is symbiotic ing strategy, and exerting effort. Feedback on affect and
in nature, these learner-technology partnerships are inher- engagement can also facilitate students in self-monitoring
ently social, even while not necessarily appearing as such. and directing time on task. One implementation of this
In the forethought phase of Zimmerman’s model (2013), feedback could be as a dashboard driven by multimodal
systems could serve as intelligent scaffolds to support both analytics (e.g., including eyetracking and gaze detection)
goal setting and task analysis. While there has been much to provide the student with intelligent feedback about
work on static technology-based scaffolds (e.g., Winne & where to both direct and focus attention, as well as relevant
Nesbit, 2009), designing adaptive and AI-driven scaffolds affective states (e.g., boredom and confusion; Azevedo &
is of current interest. For example, an intelligent agent to Taub, in press). One of the key issues, however, with dash-
support career path exploration guides undergraduate stu- boards is the importance of employing Human-Computer-
dents to visualize a large and uncertain decision space Interaction (HCI) design principles to focus attention to
where there are a multitude of choices and options (Jeon, the most relevant information and the right time. Poor
Shafran, Breitfeller, Levin, & Rosé, 2019). Taking into HCI design could easily lead to the opposite effect where
account a huge data set of options, the system provides the feedback is detrimental, leading to greater cognitive
intelligent guidance to suggest instructional and career load and distracting from the information most relevant
pathways that have the most potential for student success. for self-regulation and critical thinking.
A unique aspect of this intelligent system is that it also has Finally, in the self-reflection phase, the system can aug-
an actual human-in-the-loop to support students in goal- ment the capacity to self-evaluate. For example, consider a
setting and decision-making throughout the process while study where students were intentionally frustrated, and a vir-
also providing motivation and engagement. tual agent on the screen was designed to ‘‘watch” what was
Supporting self-efficacy and task interest/value are happening (Baylor & Rosenberg-Kima, 2006). Following
important self-motivational beliefs in the forethought the event, the agent provided an explanation for the frustra-
phase. Support to encourage self-motivational beliefs could tion, facilitating a more positive causal attribution—that the
be provided through on-screen messages or conversational frustration was due to the system rather than student error.
dialogue, for example. In particular, virtual agents can be Similarly, a dashboard could also trigger self-reflection and
392 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397

self-evaluation post-performance, delivered by a visual, a interpretations and draw inferences, evaluate situations,
prompt, or an agent. To achieve a productive self- and make decisions within the specific work context. This
evaluation, a dashboard-like intelligent interface could con- futuristic example illustrates how Corey, a professional
vey metrics about effectiveness of the performance to facili- restaurant server (or waiter), in partnership with technol-
tate a realistic assessment. The Crystal City game-based ogy can be supported to self-regulate and think critically
learning environment provides adaptive reflection scaffold- in support of his decision making before, during, and
ing tools for students based on models of reflection driven afterwork.
by learning analytics. The system facilitates reflection both This scenario (summarized below in Table 1) provides
during and following science problem solving (Taub, examples of how a technology-based system can help
Mudrick, Azevedo, Miller, Rowe, & Lester, 2017). Corey to self-regulate in each of the three phases of
Zimmerman’s model (2013). In the forethought phase,
5. Cognitive augmentation as a partnership and the role of intelligent scaffolding can help him in goal-setting and to
interface/interaction design encourage positive beliefs such as self-efficacy. In the per-
formance phase, the system can augment his cognition
In all of these instantiations, it is important to note that the and affect by providing ‘‘super-senses” through real-time
learner interacts with the technology in partnership, not as a multimodal data collection. Finally, in the self-reflection
one-way system. As aligned with social-cognitive theory, the phase, the system can analyze his performance data to sup-
partnership is designed as social in nature, even when this is port him in formulating productive evaluation and
not directly apparent (e.g., through an anthropomorphic vir- attributions.
tual agent). This type of symbiotic human-technology rela-
tionship is advantageous in the support of self-regulatory 7. Forethought phase: Restaurant server/waiter example
outcomes, as suggested by Winne and Nesbit (2009).
It is important to highlight that simply by using these In the forethought phase, the server is supported with
types of intelligent systems the learner will engage in reflec- technology-based scaffolding to facilitate decision-making
tive behavior. Accordingly, the HCI design of the interface and positive beliefs in preparation for the upcoming work
is critical so that students are directed to the most impor- shift. Here, the system will provide support for the follow-
tant information and are not overwhelmed. This includes ing functions:
design of the message delivery—whether it should be deliv-
ered through a conversational and/or embodied social  conduct task analysis to help manage the decision set,
interface (e.g., as a virtual agent representing a role or a  scaffold the process of goal setting,
perspective) or visualized through a more data-intensive  design a strategic plan for the upcoming performance,
dashboard. Unfortunately, many systems that follow the and
latter approach present too much information, with a rep-  coach positive self-motivational beliefs.
resentation similar to a busy ‘‘control room.”
Next, we provide a futuristic example showing how fea- This, in turn, will impact several self-regulatory and crit-
tures of an advanced technology system might support a ical thinking skills:
restaurant server to engage in self-regulated learning
before, during and after his work shift in order to support  Corey evaluates aspects of the upcoming shift as pro-
optimal levels of job performance. A growing global con- vided by the system as to what factors may impact the
cern is how to prepare the future workforce where individ- shift and his goal-setting.
uals will work in partnership with advanced technologies to  When Corey formulates the plan for his upcoming per-
analyze data, make interpretations and draw inferences, formance, his explanation is a form of critical thinking.
evaluate situations, and make decisions related to their  Corey makes inferences as to the rational for the sug-
specific work context. We next describe an example where gested plan, analyzing the plan and adjusting it to
a worker uses knowledge to self-regulate and critical think- achieve goals.
ing in decision making. This scenario involves a restaurant  Corey interprets suggestions and through self-talk
server, Corey, who has been a professional server (or explains to himself how he can manage a difficult task.
waiter) for 10 years, and his goal is to reach peak perfor-  Corey evaluates potential outcomes in terms of realistic
mance in his job. Here we show how he uses knowledge expectations.
to self-regulate and think critically while decision making
in the work environment. Overall, the technology design challenges for the fore-
thought phase include: (a) designing an adaptive scaffold
6. A futuristic work scenario for a restaurant server to guide the server easily through the decision-making pro-
interested in achieving peak performance cess, and (b) simulating a social relationship when ‘‘coach-
ing” the server in preparation for the shift.
With advances in technology, workers in jobs of the Conduct task analysis to help manage decision set. Imag-
future will be supported in new ways to analyze data, make ine that the system could conduct a task analysis of the
 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397 393

upcoming shift, considering situational factors (e.g., it is a

 When the server describes the plan for his upcoming performance in tan-
dem with intelligent support, the explanation is a form of critical thinking.
 Server makes inferences as to the rational for the suggested plan, analyzing

 The server interprets these suggestions and through self-talk explains to

making inferences about his performance through client responses, and

 The server makes personal decisions based on analyzing his performance,

 The server analyzes information from the system to make decisions about

 He makes inferences about a variety of factors in conjunction with his per-

 The restaurant server evaluates aspects of the upcoming shift to set goals.
holiday weekend, there is a football game, or a large dinner
reservation). This could yield possible implications based

 The server evaluates outcomes in terms of realistic expectations.

on (a) prior data for that particular restaurant; (b) the

The server interprets the evidence and evaluates his performance.

unique talents of Corey (e.g., good memory but slower;
or friendly but less-detail oriented); (c) other workers on

the best way to proceed with multiple goal structures.

duty (chef, other waiters, supervisors); and (d) contextual
themselves how they can manage a difficult task. information such as weather data. The system then would
make projections for the shift, decomposing them into
tasks; for example, the server should expect a very busy
the plan and adjusting it relative to goals.

period from 5 to 6:30 p.m. given this is immediately after

the football game, and there is also a large reservation of
15 people. By 7:30 p.m., it will be much slower than usual,
evaluating his overall progress.
Support of critical thinking skills

with an expected 75% of tables full.

sonal customer interactions.
Present options to facilitate easier goal setting through
scaffolds. Presenting this information to Corey as an adap-
tive scaffold, the system will support Corey in setting goals
for how to tackle the shift. This could include providing
him with personal options/goals such as whether he is will-
ing to work overtime if needed, versus that he has no flex-
ibility time-wise. He enters other relevant personal
information related to goal-setting, such as he is recovering
from an injury, so the system may assign him to cover the
larger tables that are closer to the kitchen where he can
 Design a strategic plan for the

 Collect cognitive and affective

ing while managing cognitive load

 Coach positive self-motivational
 Scaffold the process of goal setting

 Support real-time decision-mak-

 Conduct task analysis to help

more easily get help carrying dishes. With the support of

 Support productive attribution
 Support productive evaluation

the system, Corey evaluates aspects of the shift to set goals.

Human-technology partnership

 Collect performance data

Design a strategic plan for the upcoming performance

manage the decision set

upcoming performance
Restaurant of the future: Supporting a professional server in critical thinking to optimize performance.

through critical thinking. The system then incorporates

Corey’s input into a new plan for the day and will later
include this in the self-reflection phase (and a more com-
prehensive system may assign tasks that have less time
urgency and delegate to other co-workers where appropri-
beliefs

data

ate). When Corey describes his plan for his upcoming per-
formance during his shift, his explanation is a form of
critical thinking. The system presents a plan for the shift,
Intelligent scaffold to facilitate goal-setting and to encourage positive beliefs.

information and present suggestions; e.g., through charts, visualizations,

and Corey accepts it or adjusts the parameters, working

deliver it in a way that will lead to the most productive evaluations and
(1) designing an adaptive scaffold to guide easily through the decision-

Design challenge: what level/ type of feedback to provide and how to

in partnership with the technology to finalize the plan.

making process, (2) simulating a social relationship with technology

Augmentation by providing ‘‘super-senses” w/r cognition and affect.

To promote agency, the level of system detail can be cus-

The cyclical self-regulatory feedback loop model: forethought phase

Design challenge: designing interface to best represent important

tomized based on personal preferences to drilling down

Present performance data to facilitate productive self-reflection.

in the specifics versus just a high-level plan for the day.

Corey makes inferences as to the rationale for the sug-
gested plan. He analyzes the plan and adjusts the plan to
achieve goals.
Coaching of positive self-motivational beliefs in consider-
ation of upcoming performance. To support productive
beliefs, the system presents customized messages, such as
Facilitate orchestration of activities.

‘‘This shift will be challenging, but you can do it by focus-

ing hard until 7:30 p.m., and it will likely get easier after
that.” This feedback supports Corey in setting set realistic
expectations for success and associated self-efficacy beliefs.
text, or conversation.
System implementation

These messages could be delivered on-screen, conversation-

through ‘‘coaching”
Design challenges:

ally, or through a virtual agent serving as a simulated social

model. Corey interprets these suggestions and through
attributions.

self-talk explains to himself how he can manage a difficult

task. He evaluates outcomes in terms of realistic
Table 1

expectations.
394 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397

8. Performance phase: Restaurant server/waiter example information and automate some of the processes. It could
provide a simple overlay about the detected mood of each
Moving to the performance phase, the work shift has table, with a visualization for each of the assigned tables
begun and here the system augments the server’s ability with a face emoji indicating whether the table is happy, irri-
to sense in real-time, essentially providing him with tated, angry, or bored. The system could directly suggest
‘‘super-senses” with respect to cognition and affect. In the most important information for him to attend to
addition, the system could facilitate Corey in orchestrating (e.g., suggesting delay the delivery of appetizers to table
his activities in real-time. In this scenario, the system will one so that can he immediately attend to irate customer
provide intelligent support for the following functions: at the counter). Here, the system helps orchestrate the flow
of activities and supports Corey’s personal decision-
 collect cognitive and affective data making, especially when there are competing goals. In sup-
 support real-time decision-making while managing cog- port of critical thinking, he analyzes information from the
nitive load, and system to make decisions about the best way to proceed
 collect performance data with multiple goal structures. Through an overlay of infor-
mation or suggestions of what tasks to do next, Corey
This, in turn, will impact several self-regulatory and crit- makes inferences about a variety of factors in conjunction
ical thinking skills: with his interactions with customers and the overall cus-
tomer experience. He makes personal decisions based on
 Corey makes personal decisions based on analyzing his analyzing his performance, making inferences about his
performance, making inferences about his performance, performance through client responses, and evaluating his
and evaluating his overall progress. overall progress.
 He analyzes information from the system to make deci- Collect performance data. In the background, the system
sions about the best way to proceed with multiple goal would also track (and present where applicable) progress
structures. using metrics, such as amount of gratuity received, total
 Through this overlay of information, Corey makes infer- amount of food/drinks ordered, quickness to deliver orders
ences about a variety of factors in conjunction with his to tables, and the number of complaints. This information
interactions with customers and the overall customer is collected in preparation for the self-reflection phase.
experience.
9. Self-reflection phase: Restaurant server/waiter example
Overall, the primary design challenge for the perfor-
mance phase includes designing the interface to best repre- In this final phase, the server is provided with multiple
sent information and make suggestions to Corey through levels of feedback about his performance, related to the
data, visualizations, text, or conversation. This is all in con- original goal-setting as well as other general feedback. This
sideration of minimizing his cognitive load. could also draw from extensive data on his prior perfor-
Collect real-time cognitive and affective data. In this very mance. Here, the system will model his performance based
futuristic example and important to note would be to on data collected during the performance phase with two
address privacy issues, the customer tables could be wired objectives:
with sensors to collect multimodal data: (a) voice data (cus-
tomer tone, prosody, volume; these could indicate affective  support productive evaluation, and
states such as being relaxed, stressed, excited. The system  support productive attribution
could also collect the server’s amount of time spent talking
versus listening at each table); (b) gaze through eye- In support of critical thinking, Corey interprets the evi-
tracking (whether customers are focused on gazing at each dence provided by the system and evaluates his perfor-
other versus looking around the restaurant); and (c) per- mance. The technology design challenge for this phase is
haps even sensors to detect alcohol consumption. The ser- what level/ type of feedback to provide and how to deliver
ver, too, (hypothetically) could have biometric sensors to it, in a way that will lead to the most productive evalua-
determine his overall mood/engagement and stress during tions and attributions.
the shift, as well as his speed in completing the tasks. All Support productive evaluation. Following the shift, the
of this data could provide valuable real-time information system provides options for Corey to self-rate himself
into the feedback system to facilitate Corey in performing across different metrics, including the personal goals specif-
and making decisions on-the-job, as described. ically targeted for the shift. The system provides feedback
Support real-time decision-making while managing cogni- as to whether his assessments are realistic based on actual
tive load. Throughout the process, a key technology design data collected, to assist him in making productive self-
challenge is how to support Corey in orchestrating his evaluations. Corey is seeking to optimize performance, so
activities during the shift in a way that minimizes cognitive the feedback would be more specific than it might be to
load and makes it easier for him to make decisions. For motivate a new server. Corey reflects on his ability to work
example, his tablet interface could highlight important with customers at an optimal level with support of the
 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397 395

system and considers his performance and actions in it easier and even more enjoyable? There was huge resis-
addressing unpredictable events, multiple and competing tance in bringing calculators into the classroom in the
tasks, and the customer experience. In support of critical 1980s, and now they are considered a necessary cognitive
thinking, the server interprets the evidence and evaluates tool for mathematics students. Along this line, how might
his overall performance. we imagine intelligent cognitive augmentation as a neces-
Support productive attributions. Additionally, the system sary support of worker capacity to self-regulate and think
presents anomalies to Corey to assess his attributions for critically? This could include intelligent support while
each. For example, ‘‘It looks like you really lost energy multi-tasking or providing reinforcement during engage-
at 7:30 p.m. Something must have happened around then. ment (e.g., as detected by affective sensors).
What do you think happened?” Corey thinks, ‘‘I just was Furthermore, clear definitions are needed of critical and
very tired. I didn’t sleep well last night.” The system then analytic thinking to ensure that assessments reflect the pro-
provides realistic attributions based on data. ‘‘Actually, cesses of these definitions. This is also true for key con-
you had almost twice the volume of items delivered in structs such as metacognition, self-regulation and self-
the hour before 7:30 p.m., and you walked twice as much regulated learning where definitions appear nested in the
as usual for this time period! Great job, all considered!” literature (Dinsmore, Alexander, & Loughlin, 2008). In
Through interpretation, the server will use this information addition, researchers should link these processes to instruc-
to make productive attributions in evaluating his progress. tional strategies (problem-based learning, case-based sce-
Fostering self-regulatory functioning through feedback narios, role plays, etc.) and provide a detailed framework
loops. The cycle of learning is then completed when the of how these skills should be taught. Students’ capacities
waiter uses the self-reflection outcomes as feedback to for self-regulated learning, which is required to implement
inform future learning attempts. Feedback loops are valu- reflective judgment, underlies many of the critical thinking
able because they generate information about cognition, skill dispositions. Therefore, more studies are needed in
skills, and processes that have been shown to be malleable promoting critical and analytic thinking by engaging learn-
and modifiable through intervention and instructional ers in self-regulated learning. If the learning environment is
changes (Guthrie & Wigfield, 2000; Zimmerman (2013)). crucial to the development of critical thinking skills, what
The feedback loop is an important component to the instructional strategies can be used to promote critical
three-phase model of self-regulation because it serves as and analytic thinking?
the mechanism through which in this case, the waiter can As a result, measurement of critical thinking within the
rely on information sent through the feedback loop in self-regulated learning framework has been self-reported.
order to modify or adjust his strategies when goals were Among the most effective assessment tools that have been
not successfully achieved. For example, if Corey attributes developed over the past decade to measure self-regulatory
a low performance on the job to external and controllable processes are microanalytic protocols. Microanalytic pro-
factors such being tired, then he would be more likely to tocols target self-regulated learning as it occurs in particu-
engage in more effective planning strategies and his motiva- lar contexts and settings. Microanalysis is unique because it
tion would not be compromised. specifically targets individuals’ cognitive, motivational, and
metacognitive processes as they engage in learning or per-
10. The future of intelligent systems to optimize performance formance activities. To reflect a microanalytic approach,
by supporting self-regulation and critical thinking skills the embodied virtual agent could ask the learner to rate
their level of confidence or perception of their performance
In looking to the future, there are several areas where during a task. For example, a server having completed a
intelligent systems could further augment one’s perfor- round of customers could rate how self-satisfied they are
mance. First, AI-driven systems that collect individual per- in the way they listened to the customers requests or how
formance data over a longer period of time could learn to well they monitored the customer’s needs. The microana-
decrease cognitive load for the novice during the perfor- lytic approach embedded through technology may be a
mance phase and create a simplified version of the prob- useful assessment too for critical and analytic processes
lem. Then, the performer could ‘‘offload” some of the because it can be linked to intervention planning and job
more challenging cognitive processes and critically think performance. An individual would then be able to generate
about the problem in a specific and intentional way. In valuable information about cognition, skills, and processes
the self-reflection phase, the future holds much promise that have been shown to be malleable and modifiable
where large amounts of multimodal data can inform prac- through intervention and specific instructional methods
tice providing personalized feedback on outcomes such as (Bembenutty, Clearly, & Kitsantas, 2013; Guthrie &
focus, engagement, as well as performance. From a socio- Wigfield, 2000). Thus, microanalytic protocols may also
cognitive perspective, ideally the systems should also help learners to understand the cognitive processes that
include an element of personal agency to facilitate efficacy constitute critical thinking and use instructional activities
beliefs. that will assist learners in developing these processes.
Finally, since self-regulation can be tedious, what fea- Overall, given that researchers seem to generally agree
tures of the task might be off-loaded to the system to make that self-regulation and critical and analytic thinking
396 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397

processes can be learned through deliberative practice Guthrie, J. T., & Wigfield, A. (2000). Engagement and motivation in
(Bembenutty et al., 2013; Niu, Behar-Horenstein, & reading. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr
(Eds.). Reading research handbook (Vol. 3, pp. 403–424). Mahwah, NJ:
Garvan, 2013; Zimmerman, 2013) designing these human Erlbaum.
(student)-technology partnerships have great capacity to Jeon, B., Shafran, E., Breitfeller, L., Levin, J., & Rosé, C. P. (2019). Time-
augment self-regulation of critical thinking, and in turn series insights into theprocess of passing or failing online university
performance. Drawing from the conceptual literature pro- courses using neural-induced interpretablestudent states. Proceedings
vides new avenues to design future work environments at of Educational Data Mining. https://arxiv.org/pdf/1905.00422.pdf.
Kitsantas, A., & Zimmerman, B. J. (2006). The influence of graphing and
the human-technology frontier where humans in partner- self-evaluative standards. Metacognition and Learning, 3(1), 201–212.
ship with advanced technologies support optimal Labuhn, A. S., Zimmerman, B. J., & Haselhorn, M. (2010). Enhancing
performance. students’ self-regulation and mathematics performance: The influence
of feedback and self-evaluative standards. Metacognition Learning, 5,
173–194. https://doi.org/10.1007/s11409-010-9056-2.
References Lau, J., & Chan, J. (2018). What is critical thinking?. Retrieved fromhttp://
philosophy.hku.hk/think/critical/ct.php.
Azevedo, R., & Taub, M. (in press). The challenge of measuring processes Lombardi, D., Bickel, E. S., Bailey, J. M., & Burrell, S. (2016). High
and outcomes during learning from multiple representations with school students’ evaluations, plausibility (re) appraisal, and knowledge
advanced learning technologies. In P. Kendeou, P. Van Meter, A. List, about topic sin Earth Science. Science Education, 102, 153–177. https://
& D. Lombardi (Eds.), Handbook of learning from multiple repre- doi.org/10.1002/sce.21315.
sentations and perspectives. Cambridge, MA: Cambridge University Nguyen, K., Kallouq, B., Schuster, A., Beevers, C., Dil, N., Kay, D., ...
Press. Harris, D. M. (2017). Developing a tool for observing group critical
Baylor, A. L. (2011). The design of motivational agents and avatars. thinking skills in first-year medical students: A pilot study using
Educational Technology Research & Development, 59(2), 291–300. physiology-based, high-fidelity patient simulations. Advanced Physio-
https://doi.org/10.1007/s11423-011-9196-3. logical Education, 41, 604–611. https://doi.org/10.1152/
Baylor, A. L., & Rosenberg-Kima, R. B. (2006). Interface agents to advan.00126.2017.
alleviate online frustration. In Proceedings of the 7th International Niu, L., Behar-Horenstein, L. S., & Garvan, C. W. (2013). Do
Conference on Learning sciences (pp. 30–36), Bloomington, Indiana: instructional interventions influence college students’ critical thinking
ISLS. skills? A meta-analysis. Educational Research Review, 9, 114–128.
Bembenutty, H., Clearly, T., & Kitsantas, A., (Eds.). (2013). Self-regulated https://doi.org/10.1016/j.edurev.2012.12.002.
learning applied across diverse disciplines. A Tribute to Barry J. Nosich, G. M. (2012). Learning to think things through: A guide to critical
Zimmerman. Charlotte, NC: Information Age Publishing. thinking across the curriculum (4th ed.). Boston, MA: Pearson.
Bol, L., Hacker, D. J., Walck, C. C., & Nunnery, J. A. (2012). The effects Partnership for 21st Century Skills. (2009). Professional development for
of individual or group guidelines on the calibration accuracy and the 21st century. Retrieved from <http://www.p21.org/documents/
achievement of high school biology students. Contemporary Educa- P21_Framework.pdf>.
tional Psychology, 37, 280–287. https://doi.org/10.1016/j. Pedro, J. Y. (2005). Reflection in teacher education: Exploring pre-service
cedpsych.2012.02.004. teachers’ meanings of reflective practice. Reflective Practice, 6, 49–66.
Borge, M., Ong, Y. S., & Rosé, C. P. (2018). Learning to monitor and https://doi.org/10.1080/1462394042000326860.
regulate collective thinking processes. International Journal of Com- Phan, H. P. (2010). Critical thinking as a self-regulatory process
puter-Supported Collaborative Learning, 13(1), 61–92. https://doi.org/ component in teaching and learning. Psicthema, 22, 284–292.
10.1007/s11412-018-9270-5. Plant, E. A., Baylor, A. L., Doerr, C. E., & Rosenberg-Kima, R. B. (2009).
Brookfield, S. D. (2012). Teaching for critical thinking: Tools and Changing middle-school students’ attitudes and performance regard-
techniques to help students question their assumptions. San Francisco, ing engineering with computer-based social models. Computers and
CA: Jossey-Bass. Education, 53(2), 209–215. https://doi.org/10.1016/
Cleary, T. J., & Labuhn, A. S. (2013). Application of cyclical self- j.compedu.2009.01.013.
regulation interventions in science-based contexts. In H. Bembenutty, Rosenberg-Kima, R. B., Baylor, A. L., Plant, E. A., & Doerr, C. E. (2008).
T. J. Cleary, & A. Kitsantas (Eds.), Applications of Self-Regulated Interface agents as social models for female students: The effects of
Learning across Diverse Disciplines. A Tribute to Barry J. Zimmerman agent visual presence and appearance on female students’ attitudes and
(pp. 89–124). Charlotte: Information Age Publishing, Inc. beliefs. Computers in Human Behavior, 24(6), 2741–2756. https://doi.
Dewey, J. (1933). How we think: A restatement of the relation of reflective org/10.1016/j.chb.2008.03.017.
thinking to the educative process. Chicago, IL: Henry Regnery. Schunk, D. H., & Zimmerman, B. J. (Eds.). (1994). Self-regulation of
Dinsmore, D. L., Alexander, P. A., & Loughlin, S. M. (2008). Focusing learning and performance: Issues and educational applications. Hillsdale,
the conceptual lens on metacognition, self-regulation, and self- NJ: Erlbaum.
regulated learning. Educational Psychological Review, 20, 391–409. Sternberg, R. J., Roediger, H. L., & Halpern, D. (Eds.). (2007). Critical
https://doi.org/10.1007/s10648-008-9083-6. thinking in psychology. New York, NY: Cambridge University Press.
Durlach, P. J., & Lesgold, A. M. (Eds.). (2012). Adaptive technologies for Taub, M., Mudrick, N., Azevedo, R., Miller, G., Rowe, J., & Lester, J.
training and education. New York, NY: Cambridge University Press. (2017). Using multi-channel data with multi-level modeling to assess
Engelbart, D. C. (1962). Augmenting human intellect: A conceptual in-game performance during gameplay with Crystal Island. Computers
framework (Report No. AFOSR-3233) Menlo Park, CA: Stanford in Human Behavior, 76, 641–655.
Research Institute. Winne, P. H., & Nesbit, J. C. (2009). Supporting self-regulated learning
Facione, P. A. (1990). Critical thinking: A statement of expert consensus for with cognitive tools. In D. J. Hacker, J. Dunlosky, & A. C. Graesser
purposes of educational assessment and instruction. Millbrae, CA: The (Eds.), Handbook of metacognition in education (pp. 259–277). New
California Academic Press. York, NY: Routledge/Taylor & Francis Group.
Grushka, K., McLeod, J., & Reynolds, R. (2005). Reflecting upon Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive
reflection: Theory and practice in one Australian university teacher perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.),
education program. Reflective Practice, 6, 239–246. https://doi.org/ Handbook of self-regulation (pp. 13–39). San Diego, CA: Academic
10.1080/14623940500106187. Press.
 A. Kitsantas et al. / Cognitive Systems Research 58 (2019) 387–397 397

Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Zimmerman, B. J., & Kitsantas, A. (2014). Comparing the predictive
Historical background, methodological developments, and future power of self-discipline and self-regulation measures of learning.
prospects. American Educational Research Journal, 45(1), 166–183. Contemporary Educational Psychology, 39, 145–155. https://doi.org/
https://doi.org/10.3102/0002831207312909. 10.1016/j.cedpsych.2014.03.004.
Zimmerman, B. J. (2013). From cognitive modeling to self-regulation: A
social cognitive career path. Educational Psychologist, 48, 135–147.
https://doi.org/10.1080/00461520.2013.794676.