EMERGING EVIDENCE ON THE USE

OF BIG DATA AND ANALYTICS

IN WORKPLACE LEARNING
A Systematic Literature Review

FS
O
Lisa A. Giacumo Jeroen Breman
Boise State University Northwest Lineman College

R O 6
1
This article provides a systematic literature review about nonprofit and for-profit organizations using “big

P
data” to inform performance improvement initiatives. The review of literature resulted in 4 peer-reviewed

0
articles and an additional 33 studies covering the topic for these contexts. The review found that big data and
analytics could be used to improve selection of staff and training interventions, cut costs, improve retention,

P 2
and performance as well as meet strategic business goals, but little evidence of big data analytic methodolo-
gies actually employed in workplace training and performance improvement projects was found. Additional

IA ©
robust research with methodological descriptions are needed to reveal the extent to which these methodolo-
gies yield improved human performance results that positively affect strategic business goals and the bottom
line. Opportunities for further research are also suggested.

INTRODUCTION parts of the globe. For example, the World

Wide Fund for Nature started its blended
Charities, nonprofits, and international non- learning Conservation Leadership Program in
governmental organizations (iNGOs) recog- 1998 (Stoel, 2004), far earlier than many cor-
nized early the potential of distance learning to porate universities began seriously considering
help develop staff competencies in business online learning as an educational option. Even
critical areas. This is especially true for inter- for those organizations with fewer resources
national organizations with staff and offices than one such as the World Wide Fund for
located in various, geographically separated Nature, there exist several interorganizational

• Lisa A. Giacumo, Assistant Professor of Department of Organizational Performance and Workplace Learning, 1910 Uni-
versity Dr., Boise, ID, 83725-2070. Telephone: (208) 426-1377. E-mail: [email protected] • Jeroen Breman,
Learning and Development Consultant, Grid University, Northwest Lineman College, 7600 S. Meridian Road, Meridian, ID
83642. Telephone: (208) 888-4817. E-mail: [email protected]

The Quarterly Review of Distance Education, Volume 17(4), 2016, pp. 21–38 ISSN 1528-3518
Copyright © 2016 Information Age Publishing, Inc. All rights of reproduction in any form reserved.
22 The Quarterly Review of Distance Education Vol. 17, No. 4, 2016

initiatives to increase access to available practice of instructional designers and learning

online learning. One example is the Corner- and development project managers.
stone OnDemand Foundation, which has a dis- A systematic review of recognized aca-
tance learning course library that is freely demic publications yielded very few results
available on the disasterready.org website (specifically, one) on our primary research
(Cornerstone OnDemand Foundation, 2016). question. While there were examples that big
Another example is LINGOs, a membership- data are being employed in nonprofit organiza-
driven organization, which offers a compli- tions (e.g., the United Nations), the use of big
mentary learning management system for data in projects focusing on training and devel-
members as well as access to Last Mile Learn- opment were almost nonexistent in the litera-
ing, a distance learning library with courses on ture. Therefore, the review was broadened in
financial management, people management, scope to the use of big data and analytics in

S
and project management (Last Mile Learning,” workplace learning, performance improve-
ment, and learning and development in busi-

F
2016; LINGOs, 2016).
ness and governmental settings. This
Such nonprofit organizations are no strang-
broadening of scope yielded a greater number

O
ers to the use of data to guide relief, develop-
of documents covering our topic, and therefore
ment, fundraising, and marketing efforts. For
serves as the basis for this literature review.

O
example, in marketing and fundraising efforts, We describe this literature review as a sys-
data analysts are essential resources to deter-

R 6
tematic literature review of how big data and
mine what different audiences are interested in analytics are currently being used in institu-

P 1
(e.g., types of programs) or to measure the tionally based formal workplace learning, tal-

0
effectiveness of a specific campaign (Toal, ent development initiatives, and training
2014). In addition, large amounts of data are

P 2
departments. Additionally, we examine why
gathered in program delivery within such orga- many organizations are not yet analyzing their

IA ©
nizations. For example, humanitarian organi- own big data, and what resources are required
zations use technology to register individuals to begin utilizing big data and analytics to
affected by disasters and better manage certain improve outcomes, minimize costs, increase
services that are offered to them (Chibafa, revenue, and achieve strategic business out-
2014). Additionally they can be used to create comes for the typical organization. Finally,
end-to-end visibility in their supply chain data implications for practitioners, policy makers,
(Helios Foundation, 2014). That being the and decision makers are presented. We con-
case, we hypothesized, would it not make clude with suggested directions for future
sense for such organizations to analyze if and research.
how performance improvement, including staff
development programs, resulted in opera-
BACKGROUND
tional gains?
Having observed such examples in the
Economic Evidence that Data-Based
authors’ combined 17 years of experience
working in performance improvement, train-
Decisions Improve Business Results
ing, distance education, and capacity building As recently as 2011, Brynjolfsson, Hitt, and
initiatives in nongovernmental organizations Kim developed research methods and authored
(NGOs) and iNGOs, we were interested to a study that showed evidence of firms man-
determine to what extent learner analytics are aged with a strong data driven decision making
driving management decisions about such ini- approach have output and productivity levels
tiatives. We also were interested in determin- about 5%–6% greater than would normally be
ing the potential uses of “big data” (to be expected, given investments beyond tradi-
defined below) and analytics to improve the tional internal and external data collection,
Emerging Evidence on the Use of Big Data and Analytics in Workplace Learning 23

input analysis, and IT resources (Brynjolfsson these findings reflect a growing industry trend.
et al., 2011). Tambe and Saraf (2014) found The question is: if there are opportunities for
that organizations that employ the use of big learning and development managers and
data technologies can show up to 1%–3% instructional designers to harness the potential
higher productivity than the average organiza- benefits of big data and analytics for purposes
tion that does not use this technology, but only of increasing efficiency and effectiveness of
in those organizations that hold significant training and workplace learning efforts, what
data assets and are situated in labor markets evidence can be found through an objective,
where a cadre of employees possessed comple- systematic review of the state of scientific
mentary technical skills. It would seem that the research and professional practice to guide
analytical skills possessed by experienced per- researchers, practitioners, and students who
formance improvement specialists and instruc- are interested in acquiring these skills?

S
tional designers, coupled with proficient
software engineers, data analysts, and informa-

F
tion technology (IT) professionals, could What Is Big Data?
potentially serve as this “cadre” of employees

O
A preliminary search for content associated
possessing these technical skills.
with big data conducted in most any scientific

O
academic database will return thousands of
Big Data, Analytics, results. Mayer-Schönberger and Cukier (2013)

R 6
and Workplace Learning suggest that big data will result in “the ability

1
of society to harness information in novel

P
The use of big data and analytics in work- ways to produce useful insights, goods, and

0
place learning appears to be one of the current services of significant value” (p. 2). IT

P 2
big trends of organizational development, as a researchers are developing data analytics
review of trade magazines, professional orga- methods to use big data and to solve data-

IA ©
nization websites, and the blogs of leading related problems in contemporary business
learning and development authors and consul- organizations (Chen, Chiang, & Storey, 2012).
tants will demonstrate. The Chartered Institute Recently, governments, e-commerce, and
of Personnel and Development lists big data as health organizations have begun to systemati-
one of ten key shifts learning and development cally leverage the vast amount of web-based,
professionals need to make: “There’s more mobile, and sensor-generated data for insights
than enough information out there, but decid- leading to new science and development of
ing what to use and how to make it valuable is systems (Chen et al., 2012).
a challenge” (“Get Ready”, 2015, p. 9). While academic database search results for
However, from experience and involve- big data will return a vast amount of research
ment with hundreds of colleagues connected articles, there is nevertheless no widely
through professional societies and confer- agreed-upon, concrete definition of the term
ences, the authors have not yet personally wit- (Dumbill, 2013). One of the more popular
nessed much direct evidence of training early definitions of big data focuses on vol-
opportunities to develop skills in working with ume, velocity, and variety; the “three ‘Vs’” of
big data and data analytics, nor implement data management (Laney, 2001). In this
those skills consistently in general practice. schema, there must first be a large amount of
Young (2015) reported that as of 2014, data to be collected and analyzed, called vol-
Deloitte found that only 14% of companies ume. Velocity speaks to the timeliness of the
were conducting statistical analysis on data collection and analysis in order to maxi-
employee data. However, the companies con- mize its value. The various types of data, both
ducting these analyses report an increased semistructured and unstructured data such as
investment in analytics and modeling and audio, video, web, and text, along with
24 The Quarterly Review of Distance Education Vol. 17, No. 4, 2016

traditional structured data constitutes variety. have generally been employing smaller and
More recent discourse about big data indicate a less complicated datasets. We suggest, there-
fourth “V,” or the need to realize the hidden fore, that a beneficial definition of big data is
value (Gantz & Reinsel, 2011). Chen, Mao, the ethical collection of data points from a vol-
and Liu (2014) posit that this is the more ume of learners in a given population over
important piece of handling big data, being enough time to show patterns or changes in
able to uncover the value of large-scale data patterns, with the velocity required for timely
sets. However, researchers agree that the defi- policy, program, or organizational manage-
nition for big data may adapt to the different ment decisions, that come from a variety of
needs of researchers across diverse disciplines both primary and secondary sources, intended
and will also continue to evolve as technology to result in added-value in the learners’ or
evolves (Chen et al., 2014; Dumbill, 2013; organizational performance. We suggest that

S
Gantz & Reinsel, 2011; Manyika et al., 2011). this definition would be of use to researchers
While the four “Vs” of big data schema can

F
and analysts serving individuals’ and organiza-
be used to define current research endeavors in tional needs in traditional education, training,

O
traditional education, training, or workplace informal workplace learning environments, or
learning setting, it should be noted that a gap other noninstructional performance improve-

O
exists within the literature reflecting the extent ment interventions.
to which most researchers and practitioners in

R 6
these areas have access to data points, technol-
Why Are Analytics Important?

1
ogy resources, and the human resources

P
required to process vast quantities of informa-

0
tion. While some of this access is related to Analytics are the methods that specialists
use to derive meaning from the raw big data

P 2
management culture, costs, and capacity, pri-
vacy is also more protected in an employer- and rich layers of information available to

IA ©
employee or business-business relationship inform data-based decisions (Gandomi &
than in a market-consumer relationship and Haider, 2015). These methods are a process of
therefore inherently limits the amount of infor- gathering, cleaning, organizing, and analyzing
mation actively collected for decision-making. data available from multiple databases to
Therefore, the sheer amount of information answer questions about employees’ perfor-
that is considered big data in a Fortune 500 mance, social network and community of prac-
company is not necessarily aligned with what tice participation, training interventions, use of
is considered big data in a more traditional online learning resources, e-learning, and other
education, workplace learning, training, per- electronic learning systems. The term “learn-
formance improvement context. ing analytics” has been used when researchers
For example, to some researchers in infor- investigate workplace learning or instructional
mation management, big data is often defined systems. Bersin (2013) calls this “datafication”
in tens to hundreds of terabytes of information process talent analytics or people analytics in
pulled from a variety of sources, resulting in order to make a connection with professionals
data sets that are so large and complex that tra- supporting talent development. Others have
ditional processing software is not robust called this process HR analytics or intelligent
enough to extract meaningful and valuable analytics (Pape, 2016; Young, 2015). The pre-
information. Meanwhile, the volume and vari- cise descriptive term placed in front of the
ety of data that has been described by individ- word analytics can be customized to meet the
uals conducting big data research and expectations of one’s target audience. These
practitioners working in educational technol- analytics can be used to answer desired ques-
ogy, learning sciences, human performance tions with a descriptive, predictive, or even
technology, and human resource development, present prescriptive recommendations.
Emerging Evidence on the Use of Big Data and Analytics in Workplace Learning 25

Workplace Learning find evidence of the use of big data in relation

to learning and development within organiza-
For the purposes of this article, we will use tions to direct further research efforts (Petti-
the term “workplace learning” to describe the crew & Roberts, 2008). A protocol and steps
primary context of interest. The term work- taken to conduct the literature review process
place learning encompasses all efforts made by was carefully designed to extract the scholarly
professionals of various roles in organizations, information available on the topic. The method
who support the strategic performance employed is explained in the sections below.
improvement, training, learning and develop- While conducting the literature review, we
ment, and talent development needs of adults. found only one scientific, academic study,
Today, these functions are most frequently which reported the use of big data and analyt-
supported by providing employees or volun- ics in iNGOs’ internal training, learning and

S
teers with access to electronic performance development, or performance improvement
support systems, e-learning, online learning,

F
initiatives. Because a single source cannot rea-
and other types of talent development support sonably be used to answer a research question
that are traditionally associated with distance

O
in a literature review, research question one is
learning in NGOs, not-for-profit organiza- not answered in this study.

O
tions, governmental, and military organiza-
tions, as well as for-profit organizations.

R 6
Search Strategy

P 1
RESEARCH QUESTIONS As part of the systematic search approach

0
used in this study, we identified major terms
arising from the research questions, context,

P 2
In order to determine to what extent big data
and analytics are being used by organizations and associated variables. In addition, syn-

IA ©
and review the scientific research available to onyms and related terms were tested for their
support decision makers, policymakers, adop- ability to retrieve desired literature. Those
tion by practitioners, this work investigates the terms that did not add relevant articles were
following questions: then excluded. We used the Boolean OR to
include synonyms and the Boolean AND to
• Question 1: Does the use of big data and link the major terms with the desired contexts
analytics either reduce costs or improve and fields of study. Primary sources were also
iNGOs’ training efficiency, workplace checked for other supplementary references
learning effectiveness, and performance they might provide.
outcomes?
• Question 2: What training and workplace Search String
learning outcomes, can be informed by the
use of big data and analytics? The terms used in the searches in this
• Question 3: What factors are involved with review included:
an organization’s ability to leverage big
data and analytics for training and work- “big data” OR “analytics”
place learning?
These were searched in conjunction with terms
for workplace learning and for social sector
METHODS organizations. The search string used for the
search by topic in the Web of Science and by
We implemented a formal, systematic litera- subject in the Academic Search Premier, for
ture review process of academic literature to workplace learning was:
26 The Quarterly Review of Distance Education Vol. 17, No. 4, 2016

training OR “workplace learning” OR bases searched included those identified as rel-

“human performance improvement” OR evant for social sciences, learning and training
“learning and development” in the workplace, governance, business and
not-for-profit organizations, including: Aca-
The query for social sector organizations demic Search Premier, Applied Science &
included: Technology Full Text (H.W. Wilson), Busi-
ness Source Premier, Education Research
“nonprofit” OR “not for profit” OR “non Complete, Environment Complete, ERIC,
governmental organization” OR “non gov- Family & Society Studies Worldwide, Green-
ernmental organization” OR NGO
FILE, LGBT Life with Full Text, Library Lit-
erature & Information Science Full Text (H.W.
The combination of all three sets of terms
Wilson), Military & Government Collection,
yielded only one result in only the Academic

S
Philosopher’s Index, Professional Develop-
Search Premier database search from a non- ment Collection, PsycARTICLES, PsycINFO,

F
peer-reviewed journal. Therefore, as previ- Psychology and Behavioral Sciences Collec-
ously stated, we responded by deleting the tion, Public Administration Abstracts, Race

O
context restriction for social sector organiza- Relations Abstracts, Small Business Reference
tions, and subsequently focused on reviewing Center, Social Work Abstracts, Vocational and

O
the available literature on big data and work- Career Collection, Web News. Because only
place learning.

R 6
four relevant peer-reviewed articles were
The Boolean NOT was used to help limit returned from these databases, we next broad-

P 1
the search from retrieving results that were not ened the search scope to include magazines

0
directly related to the use of big data, analytics, and trade publications. This alteration in scope
and workplace learning. These excluded terms

P 2
returned an additional 33 results.
returned research literature that was related to

IA ©
computer science, training of data scientists,
computer chip development, bioinformatics, Search Process
and advanced medical specialists. Addition-
The searching of the digital libraries was
ally, searches were initially limited to refereed
conducted by the two authors, who were also
peer reviewed academic journals.
the primary reviewers of the collected infor-
mation. The authors collaborated on develop-
Resources Searched ing the search strategy and search terms. The
full list of articles identified by the searches
The search of resources was initially con- was then further refined according to a set of
ducted using two digital libraries that each inclusion and exclusion criteria. Table 1
search different combinations of written depicts the total number of publications
resources, the ISI Web of Science and Aca- returned for each library search conducted.
demic Search Premier. In the ISI Web of Sci- The relevant number and the percentage of the
ence, the core database was searched. In the total relevant returned publications from each
Academic Search Premier, the electronic data- library is also shown.

TABLE 1
Library Search Results

Library Relevant Not Relevant Total Percentage Relevant

Web of Science 4 197 201 1.99%

Academic Search Premier 33 346 379 8.71%
Emerging Evidence on the Use of Big Data and Analytics in Workplace Learning 27

Inclusion Criteria mance improvement journals. These results

are summarized below according to each
Articles that described examples of the use
research question. As mentioned previously,
of big data and analytics to inform workplace
we were not able to answer research question 1
learning initiatives and outcomes associated
(“Does the use of big data and analytics either
with its use were included in this review. This
reduce costs or improve iNGOs’ training effi-
literature base includes publications, technical
ciency, workplace learning effectiveness, and
reports, or other “gray” literature that
performance outcomes?”) due to a lack of
describes empirical studies or other study
available scholarly literature.
designs in which big data or analytics were
applied in the primary workplace learning con-
text of interest. To be included in this review, Question 2: What Training
workplace learning was required to be the pri- and Workplace Learning Initiatives

S
mary context of the study. Can Be Informed by the Use

F
of Big Data and Analytics?
Exclusion Criteria

O
Staff Development Intervention
Articles that described examples of big data Selection, Candidate Selection,

O
and analytics used in educational organizations,
and Retention
such as schools for children, colleges, universi-

R 6
ties, and other higher education organizations, Many of the initiatives that can be better

1
were excluded from the analysis. These studies

P
informed by the use of big data and analytics
were excluded because assessment of learning

0
center on a more granular, efficient, and effec-
is often focuses on academic performance that tive staff performance development and reten-

P 2
may not find direct application beyond the aca- tion. Big data and analytics allows for better
demic setting. Also, the needs of the audience, pattern recognition and identification of high

IA ©
as well as the information gathered from the performers and critical behaviors when they
analytic tools used by educational organiza- are recorded by employees, managers, and
tions, generally serves purposes quite different online user navigation information (Dutton,
than the context of noneducational businesses, 2014; Gardner, McGranahan, & Wolf, 2011;
who are developing adult abilities (i.e., to pro- Hall, 2013; Paine, 2015; Young, 2015). These
duce measured improvements in efficient and individual patterns can also be aggregated to
effective work output resulting in cost savings reveal highly localized staff retention distribu-
or revenue increases associated with strategic tions, which can be modeled to inform organi-
business outcomes). In addition, articles were zational management decisions (Bersin, 2013;
excluded from the search results when they Freifeld, 2014; Hyatt, 2009; Roberts, 2007).
were focused on machine learning, machine For example, big data and analytics can be
pattern recognition, data science or computer used to generate information valued by chief
science, training data scientists, railway safety, executive officers, chief financial officers, and
genetics, biotechnology, medicine, bioinfor- learning and development managers about
matics, predictive marketing analytics, and workplace learning interventions and
conference announcements. resources that employees seek out, those that
are used most often by employees and their
RESULTS managers, and those that each would recom-
mend to others (Bersin, 2013; Everson, 2015;
The results of this literature review confirmed Freifeld, 2014; Gardner et al., 2011; Hall,
that a surprisingly small amount of work has 2013; Hartley, 2013; Kettleborough, 2014;
been published in scientific academic journals Paine, 2015; Pease & Beresford, 2013; Tambe
on this topic, including training and perfor- & Saraf, 2014). With innovative performance
28 The Quarterly Review of Distance Education Vol. 17, No. 4, 2016

development software, learning and develop- cialized expertise to quickly solve problems
ment specialists and managers can track (de Laat & Schreurs, 2011; Dutton, 2014).
employee actions, contributions, and output Such technology can also track performance
results, post intervention (Baldassarre & behaviors online and, more recently, offline, as
Finken, 2015). Over time, a selection of inter- well. De Laat and Schreurs (2011) developed
ventions, by design, type, and content, can be an online tool focused on real and urgent prob-
balanced against associated costs and produc- lems in the workplace that require learning
tivity through the use of big data and analytics. relationships with colleagues or known
As more individuals are tracked, and as more experts, and combines three streams of infor-
data is collected on each individual within the mation: (a) an overview of current problems
organization, intervention selection and asso- on which professionals are currently working;
ciated productivity could eventually be pre- (b) a network visualization based on existing

S
dicted by comparing similar profiles, professional relationships; and (c) the organi-

F
backgrounds, and workplace learning opportu- zations or subdivisions within the greater orga-
nities, in combinations that prove to have sig- nization in which these network members are

O
nificant influence on performance outputs located. Through investigation, another social
(Bersin, 2013; Pease & Beresford, 2013).
media type of learning software (Siadatya,

O
Gašević, & Hatala, 2015) found that recom-
Proposals for Human Capital Investments

R 6
mender system technology aids discovery of
Can Be Translated in Business Numbers useful performance development opportuni-

P 1
Talent development and performance ties, and underscores the critical need for

0
improvement professionals can leverage big seamless social media integration in modern

P 2
data and analytics to help them develop stan- workplace learning environments (Everson,
dard human capital metrics, indicators, and 2015). The “design of work and learning envi-

IA ©
scorecards. Human capital metrics, indicators, ronments needs to be intertwined” so that
and scorecards can then be used to describe the learners can “easily share their experience with
cost and added value of talent investments in their colleagues” (Siadatya et al., 2015, p.
the language of business numbers such as 1017).
money, cost, and returns on investment (cf. When social network information is cou-
Berk, 2004, 2005; David, 2006; Dell, 2012; pled with workplace performance, actionable,
Dutton, 2014; Hartley, 2013; Higgins, 2014; immediate feedback about progress, miscon-
Kristick, 2012; Moore, 2005, 2011; O’Leon- ceptions, and knowledge and/or skills gaps can
ard, 2012; Pease & Beresford, 2013; B. Rob- promptly be communicated between managers
erts, 2007; P. Roberts, 2015; Saunderson, and employees. Training and performance
2014; Tozman, 2012; Young, 2015). Higgins improvement specialists, too, as well as execu-
(2014), for example, describes an organiza- tives, can use such information to make deci-
tion’s investment in promotions to develop sions about intervention selections and
internal employees’ competencies cost the
organizational management or development
organization 30%–35% less than the hiring of
strategies. For example, Dutton (2014)
external candidate replacements.
reported that leveraging social networks and
big data analytics in a change management
Social Networks and Big Data Analytics
project resulted in a $25 million savings in 6
Support Real Time Performance months for NWH Global. For Halliburton,
and Productivity Changes there was a 10% productivity increase, 22%
Social network technology can be used to revenue increase, and a 66% cost reduction
connect individuals with those who have spe- within a 9-month period.
Emerging Evidence on the Use of Big Data and Analytics in Workplace Learning 29

Question 3: What Factors Are Involved together, and ultimately, unable to handle big
With an Organization’s Ability data. Computer data scientists provide support
to Leverage Big Data and Analytics with accessing and merging data streams to
for Training and Workplace Learning? develop useful reports and data visualization.
They accomplish this work with specialized
The resources, culture, and talent or skills analytics software tools and dashboards, some-
that an organization possesses make up the times acquired off the shelf or in the “cloud,”
three greatest factors that can impede any other times developed in-house. These analyt-
efforts to build an analytical organization ics tools are needed to process the large
(Dutton, 2014). As most organizational leaders amount of data that are merged from a number
will state, employees are an organization’s of data streams, each organized and structured
most valuable assets (Higgins, 2014). How- differently, which results in a highly complex

S
ever, the resources and culture can make or network of information (Berk & Magee, 2005;
break the move to an organization that Dolezalek, 2003; Dutton, 2014; Hartley, 2004;

F
embraces data-driven decision-making. Moore, 2005). The massive amount of data
assets that track employees’ performances and

O
resource consumption require considerable
Resource Assets Including Data storage space as organizational systems are

O
Analysis Expertise, Analytics Software, built and as the organization continues to grow
and Data Storage

R 6
(Berk & Magee, 2005; Hall, 2013; Moore,
2005). A strong IT department can help main-

1
An interdisciplinary team that can work

P
tain these types of systems, as well as help

0
together to leverage data assets is required to
ensure privacy of both the data and the analysis
derive value from big data. This team can get

P 2
results.
started by assessing the analytical capabilities
of key learning and talent development leaders

IA ©
or human resources managers to determine Culture of Data-Based Decision-Making
their ability to acquire and manage the flow of Aligned with Business Goals, Use of
big data as well as to derive value from it (Dut- Metrics, and Performance Management
ton, 2014; Hagel, 2012). O’Leonard (2012)
advises that if the learning and development or Dutton (2014) notes that if data is not
human resource leaders in an organization do employed to make better decisions, then an
not possess such analytical capability, then increase in an organization’s analytical abili-
these can be either developed or additional tal- ties will not significantly improve organiza-
ent can be recruited. Paine (2015) describes tional performance. In such cases, an
three case studies in which at least three key important cultural change may be required.
individuals, each with a specialty in one of the Directors, who historically made decisions
following backgrounds would support this based primarily on intuition, conversations, or
effort: workplace learning analytics, computer other existing knowledge, may be required to
data scientists, and an IT specialist with expe- make changes in their practice by tapping into
rience in big data management. the massive data stream provided by big data
Talent with workplace learning and analyt- and its analysis in order to deliver evidence-
ics expertise can lead the effort to ensure that based decision-making, as well as to measure
the right data are measured and they are mea- progress towards meeting strategic business
sured accurately. While Hagel (2012) and objectives and performance development
Hartley (2004) note that analytics can be pow- goals (Paine, 2015). Organization leaders are
erful business tools, enterprisewide human also an integral component to cultural change,
resource management systems are all too often and can support the cultural change by com-
dated legacies with different pieces layered municating value for an analytical initiative
30 The Quarterly Review of Distance Education Vol. 17, No. 4, 2016

and fact-based decisions that are aligned with peer organizations, to get a complete picture of
business goals and made with confidence how an organization is performing, and mea-
(Hagel, 2012; Young, 2015). sure the results of management decisions.
Along with this focus on decision-making
aligned with business goals, Toterhi (2014)
Responsive and Responsible Leadership;
describes the importance of developing and
using human resource metrics and scorecards From the Top Executives
with directional indicators and corresponding to Line Managers
actions based on careful detailed analysis and
Most of the articles reviewed in this study
conscientious interpretation. According to
suggested that learning and development, and
Dolezalek (2003) and Toterhi (2014), this type
human resources professionals would need to
of practice and analytic information can be

S
use value metrics and learn to speak the lan-
drawn from different sources to help demon-
guage of finance and become proficient in

F
strate the value of training opportunities. For
business acumen (Hall, 2013; Higgins, 2014;
example, the utilization of analytics applica-
Hyatt, 2009). While this is sound advice and

O
tions with data inputs including workforce
can be recommended to all those who have not
metrics and training usage provides better

O
yet embraced the communication across orga-
insight when trying to evaluate whether or not
nizational silos in a language that incites
a training is correlated with improved perfor-

R 6
mance and it is more cost effective than tradi- results from the finance and executive leader-

1
ship, these individuals also need support in

P
tional needs assessment and evaluation
developing such skills. In addition, finance

0
practices. Additionally, Baldassarre and
Finken (2015) suggest managers at all levels professionals will also need to alter reporting

P 2
can make better use of the real-time data to practices to acknowledge the value of invest-
ment into human capital assets.

IA ©
garner results by engaging in developmental
check-ins with their direct reports more often When data is integrated from multiple sys-
than the formerly prescribed once or twice per tems, including human resources, budgeting,
year routine performance management accounting, learning databases, and employee
approaches. Brief but routine check-ins with information, as noted by Dutton (2014), exec-
employees, spanning from the front-line utives should be able to benchmark that data
employees through upper management, should against surveys, courseware, and facilities, and
be conducted to review successes, barriers, and then determine how individual courses affect
to acknowledge changes in behaviors as well the organization’s performance. However,
as add value and provide additional motivation based on our experience, line managers’
for continuous performance improvement. responsibility for noting training participation
Lastly, Wroe (2012) points out the impor- and outcomes in human resource systems is
tance of questioning internal data and bench- not yet routine. Without a responsible role for
marking against that data. This becomes monitoring participation in training and work-
possible when external benchmarking data is place learning programs, large gaps of data
included for comparison with best-in-class form and accurate findings may prove to be
peer organizations in analysis methods. Big elusive. Therefore executive leadership may
data can support better executive decision- need to communicate these expectations for
making by facilitating the combination of monitoring and embrace an analytical culture,
often isolated datasets, which include informa- balancing data with interpretation and qualita-
tion on performance data, assessment data, tive stories, to build organizational enthusiasm
recruitment data, recruiter ratings, manager for data-based decisions (Toterhi, 2014; Paine,
ratings, and manager performance, as well as 2015).
Emerging Evidence on the Use of Big Data and Analytics in Workplace Learning 31

DISCUSSION path forward for organizations (Dutton, 2014;

Hagel, 2012; Paine, 2015; Young, 2015). We
This section includes a discussion of what we believe this to be true due to the dearth of
learned, including the nature of the limited lit- descriptions of methods applied and evidence-
erature base focused on big data, analytics, based results in the literature. Without robust
workplace learning, and performance evidence of many success stories stemming
improvement, as well as what we can learn from validated and replicable results, it is diffi-
from existing, related big data and analytics cult to promote change as the “what’s in it for
research in computer science, business, and me” becomes a greater risk. Additionally, we
economics. We now examine each of these believe this culture shift requires more than
elements and discuss why we believe these overcoming senior organizational executives’
findings to be the case, and how these findings (chief executive officer, chief financial officer,

S
might impact organizational learning. We will chief operating officer, chief learning officer,
also make suggestions on how to overcome etc.) perceptions of the potential investment

F
implementation barriers, based on the system- risk and rewards.
atic literature search findings and our profes- Such efforts also require upper manage-

O
sional experience. In addition, current ment to champion changes to organizational
systems, processes, and actual work tasks.

O
literature related to equipping organizations
with the resources required to leverage the Systems and process changes require front-line

R 6
benefits that can be derived from the use of big professionals to be entering the workplace
anticipating projects in this area and equipped

1
data and analytics is also covered in this dis-

P
cussion, as this information was deemed to address them. As Manyika et al. (2011)

0
imperative for policy makers, decision makers, echo, we believe that there is an urgent need to

P 2
managers, thought leaders, and advanced prac- prepare the next generation of instructional
titioners in workplace learning and distance designers, human performance improvement

IA ©
education. specialists, and human resources development
specialists to collaborate on multidisciplinary
teams to identify, collect, and analyze multiple
Systematic Literature Review complex streams of data for valuable informa-
tion in order to inform data-based decision
As previously mentioned, the results of this
making that either reduces operational costs or
literature review reveals very little scholarly,
increases productivity. This would likely
peer-reviewed evidence to illustrate the bene-
involve research grant funding and industry
fits of using big data and analytics to achieve
partnerships, where faculty researchers partner
strategic business objectives, reduce costs, or
with organizations to acquire anonymized data
otherwise improve human performance.
streams to analyze, cocreate new systems,
Indeed, there are but a handful of articles avail-
technology, and mentor teams of interdisci-
able in popular training and talent develop-
plinary participants who are given an opportu-
ment industry magazines and trade journals to
nity to apply new methods, investigate models,
suggest potential benefits that human
and share the results in academic publications.
resources, training department managers, and
performance improvement consultants might
enjoy if an organizational shift in culture was Current Recommendations
undertaken. We agree with several of the and Comparisons With Related Work
authors we cited asserting that changes in in Computer Science and Business
organizational culture surrounding decision-
making, away from being based primarily on Researchers working for McKinsey Global
experience and intuition with more emphasis Institute predicted that by 2018 there would be
on massive information inputs, is part of the a shortage of 140,000 to 190,000 people
32 The Quarterly Review of Distance Education Vol. 17, No. 4, 2016

possessing deep analytical skills, and a short- growth outputs. However, there appears to be a
age of 1.5 million data-savvy managers with “missing link” in the research literature that
the ability to analyze big data for the purpose specifically investigates the changes in pro-
of making expedient decisions (Manyika et al., ductivity and growth that firms realize from
2011). Learning and development analysts investing in IT. This is similarly true for other
with the skills to move beyond operational performance support initiatives related to mea-
reports and be capable of answering questions suring upgrades to performance support sys-
about how spending is being allocated in dif- tems, talent development initiatives, training
ferent areas of an organization and its relation- interventions, and other performance improve-
ship with career progression, performance, ment interventions.
retention, and product innovation, would better Off-the-shelf big data analysis methods and
benefit decision making in organizations (Ber- applications for learning and development or

S
sin, 2013). In the past, the ability to understand performance improvement initiatives are not
how investments in an organization drives currently widely available. Increasingly there

F
results has been difficult to demonstrate sim- are cloud-based services where very large data
ply from satisfaction ratings collected in work- sets are uploaded to the cloud for analysis,

O
shops. However, such information can be now such as SkillSoft SumTotal, SAP SuccessFac-

O
be discovered with the rich data layers cur- tors, Saba Analytics, and Oracle Taleo. How-
rently available to organizations through big ever, when combining information sources

R 6
data. This means that it is now possible to make from legacy systems, as well as other cloud-
based systems, bespoke (i.e., custom made-to-

1
a shift from linking specific, one-off or even a

P
series of training interventions directly to indi- order) software is often required before data

0
viduals’ performance, historically a difficult from all corners of an organization can be

P 2
connection to establish. The layers of data merged, analyzed, and reported. Creating
from groups of employees’ involvement in bespoke analyses and applications are costly,

IA ©
social network development and communities and require advanced computer science and IT
of practice in an organization may be used to skill sets. For example, key challenges in the
build personas of successful individuals, and development of big data applications include
thus recommendations for continued develop- data representation, redundancy reduction and
ment, all of which can be tested against lon- data compression, data life cycle management,
gevity and career progression in an analytical mechanism, data confidentiality,
organization. But this ability is only possible if energy management, expandability and scal-
an organization chooses to access or develop ability, cooperation of interdisciplinary
individuals with the knowledge and skills to researchers to complete analytical objectives,
collaborate on the storage, retrieval, and analy- storage capacity, database design, and security
sis of the data for valuable meaning. (Chen et al., 2014).
To gain value from using big data, one The rich data layers from structured busi-
needs “big information technology,” as well as ness data will provide better insight into the
computer science resources. In the early contextual clues about the results of perfor-
2000s, researchers started to show the value of mance improvement opportunities offered to
productivity and output growth for firms that employees through distance learning and edu-
invest substantial resources into IT (Brynjolfs- cational technologies. These include:
son & Hitt, 2003; Daveri & Mascotto, 2006;
Kim, Lin, & Simpson, 2015; Matteucci, • surveys;
O’Mahony, Robinson, & Zwick, 2005). There- • demographic information;
fore, we now understand that firms and market • semistructured data from project manage-
sectors with vast IT resources and significant ment reports;
budgets realize increased productivity and • employee profiles;
Emerging Evidence on the Use of Big Data and Analytics in Workplace Learning 33

job performance data; and productivity, performance, retention, promo-

• unstructured data, such as internal social tions, and change in costs and billable outputs.
media, computer usage, access of internal While these methods have been used for e-
documentation, participation in profes- commerce, market intelligence, e-government,
sional development, calendar entries, et and politics, it is a small leap to consider how
cetera. these very techniques could be further lever-
aged for training and performance support sys-
In addition to learning more about the con- tems design as well as personalized training
text and how to better design and select perfor- recommender systems. For example, potential
mance support systems and allocate training outcomes of these analytic methods for all
resources to individuals who will endeavor to stakeholders include improved transparency,
grow their skillsets, the business value and participation, and satisfaction, resulting from a

S
return on investment will be easier to assess better fit between employees’ innate interests,

F
when big data methods are employed. After
abilities, and performance with improved
the performance support system is imple-
selection of performance improvement sup-

O
mented, big data can provide better insight into
ports that are valued and validated. Managers
the increased productivity and output realized
could then feel empowered and able to match

O
as a result of interventions. The realization of
employees’ interests and optimal performance
patterns of optimal workplace performance for

R 6
individuals will allow managers to provide areas with performance support systems opti-
mized for efficiency and effectiveness, along

1
individualized work structure and assignments

P
that are best suited to abilities and interests. with training programs, that have been recom-

0
Doan, Ramakrishnan, and Halevy (2011) mended according to increased productivity

P 2
noted that Web 2.0 content, which was being and more transparent human resource patterns.
authored by users on various forums including

IA ©
newsgroups, social media platforms, and Limitations to the Findings
crowdsourcing systems, offers insight on dif-
of This Literature Review
ferent stakeholder groups’ attention and focus
through information integration for interested
researchers and data analysts. For example, as General limitations
Bo and Lillian (2008) note, text analysis and While research exists to document the bar-
sentiment analysis techniques are frequently riers to adoption, and while these barriers need
employed to garner information about opin-
to be considered, additional scholarly evidence
ions for social media analytics. Analytic tech-
is needed to validate effective methodology
niques have been developed to mine
and expectations for financial value. The use
association rules, discover database segmenta-
of big data and analytics is not a direct path to
tion and clustering, detect anomalies, and mine
graphs, all of which can produce information performance improvement, however, and thus
about target product use and users’ opinions does not guarantee benefit to an organization.
(Adomavicius & Tuzhilin, 2005). Similar data Big data’s integration into an organization's
analysis techniques could be applied to organi- practices is advocated as a way to improve
zations’ internal electronic dialogue tools and cost-effective resources strategic decision
when compared with employee roles and back- making to recruit, retain, and maintain
grounds, performance records, and profes- employee productivity and engagement. Addi-
sional development activities, which yield tionally, to date, its suggested use, as well as a
important information on morale, departmen- very small number of case studies, may be a
tal focuses, skillsets at different levels of the surrogate for measures of perceived potential
organization, reactions to training, patterns of value.
34 The Quarterly Review of Distance Education Vol. 17, No. 4, 2016

Threats to Validity ature uncovered in this systematic review, it is

clear that a consensus is building: such data
It may be possible that not all relevant pub- can be used at all levels of capacity develop-
lications were identified. The extent of the ment, performance improvement, and perfor-
findings are based on the search criteria used, mance support, from societal policymaking to
as well as the limitations of the research data- organizational decision-making. This data can
bases harnessed for this project. No preexist- also be used towards the development of indi-
ing known set of references were used to viduals.
validate the search terms, and, as described The methods used to gather and analyze big
previously, the search was broadened as addi- data can be used to improve learning environ-
tional terms were suggested in articles. The ments, training outcomes, and the experiences
combination of searching through several data- of individual users. A tight economy, which

S
bases relevant for the fields of study, in addi- has been very slow to recover from a severe
tion to searching references of included studies economic crash in 2008 and decreased budgets

F
along with the use of Google Scholar, would for training efforts, has perhaps stalled a more
suggest that all relevant work was identified.

O
widespread development and integration of
Additionally, future work published from 2016
these methods. Yet, we know that at this time
and forward is not included in this work.

O
the opportunity to capture and analyze data
Lastly, publication bias is also a potential
from educational technology and electronic

6
threat to validity. It is possible that scholars

R
performance support systems is growing rap-
have found negative results and then not pub-

1
idly, and organizations that have mastered a

P
lishing their findings, leaving those findings
culture of continuous improvement can use

0
invisible to the current study. (This could
such tools to help survive periods of economic
include practitioners who are not publishing

P 2
downturns.
their work in scholarly venues, and have devel-
Business and organizational leadership

IA ©
oped case studies with various results that are
needs to invest in the technology and the asso-
not as widely shared.)
ciated human resources that will enable tap-
ping into the wealth of data that is available
IMPLICATIONS today. Leadership needs the technology to col-
lect, store, process, and report on data. Ideally,
Those who are formally trained to engage in an this process should be automated as much as
evidence-based professional practice in roles possible. The quality of the data, and accuracy
associated with the fields of education, learn- of it, too, are essential, given the life-changing
ing, human resource development, training, decisions that will potentially be based on it.
human performance technology have been uti- With that requirement come the parallel issues
lizing various methods of data analysis to of privacy and anonymity: it should not be pos-
gauge learners’ newly acquired knowledge, sible, for example, to reconnect data with indi-
skills, and attitudes that were signaled by vidual employees. Such data need to be stored
changes in behavior long before computers securely as well, so that it is not accessible by
were widely available. What is a new and parties who are not authorized to see and
emerging area of practice for researchers and examine it.
practitioners are the analytic approaches Systematic checks to verify the quality of
afforded by digital big data collection and the data gathered, a healthy understanding of
improved processing power. In concert, these the difference between correlation with and
two affordances support the potential for finer causation, objectivity in reporting the data,
grained analyses and a richer understanding of along with keeping the data anonymized and
human performance for improved policy, pro- secured are all key aspects of the responsible
gram, and decision-making. Based on the liter- use of big data for decision-making (Kettlebor-
Emerging Evidence on the Use of Big Data and Analytics in Workplace Learning 35

ough, 2014; Pease & Beresford, 2013). The requests in a manner that is understood by both
analyses and conclusions that are possible with chief executive officers and chief financial
large data sets can produce reliable informa- officers. Those who are hired to carry out the
tion only when the data is correct. While this learning and development function in organi-
information can be useful to make organiza- zations, such as line managers, need to learn to
tional decisions based on statistics derived ask the right questions, analyze the relevant
from a large group of individuals (e.g., what data to help answer those questions, and man-
type of training to fund, or skillsets required age others who are learning to do it as well.
for specific roles), we do not recommended Finally, learning and development analysts
using correlational statistics to make predictive need to be brought into the organizational
decisions about specific individuals without structure, either through training, or by hiring
personal conversations between managers and (O’Leonard, 2012). Ideally, these analysts

S
their direct reports. Statistics are aggregated have been trained to analyze social networks,
observations of large groups of data points and

F
career progression, use of electronic perfor-
may not be applicable to all individuals’ spe- mance support systems, and participation in
cific needs, as they are not derived with con-

O
communities of practice, in addition to track-
sideration of all potentially available ing investment in learning and development

O
information. For example when Google training opportunities. These learning and per-
decides you are a prime target for specific mar- formance data analysts should be able to speak

R 6
keting information, it has been shown that you two languages: that of the learning and devel-

1
are likely to be a consumer but does not mean opment field, and that of executive decision-

P
you will purchase, use, and benefit from each makers and financial experts (Higgins, 2014).

0
product determined to align with your con-

P 2
sumer profile. Therefore, analytic recommen-
dations derived from available, comprehensive FUTURE RESEARCH

IA ©
digital records can be useful starting points for
systematic management conversations about Although the term big data is a common busi-
hiring practices, individualized recommenda- ness term, there is very little published man-
tions for professional development opportuni- agement scholarship that tackles the
ties, and other systemic noninstructional challenges of analyzing big data and the subse-
performance improvement decisions. Further, quent decision-making practices that such
to facilitate a culture of learning and continu- analyses could afford (George, Haas, & Pent-
ous improvement, it is important to report what land, 2014). The lack of reliable, rigorous
did not work as much as what did work, for research information suggests that workplace
often more is learned from failure than from learning researchers have not yet even reached
success. the early adoption phase of Rogers’ (2003)
While leadership will need to develop the diffusion of innovation model. Additional rig-
skills to interpret the data that are presented to orous research and interdisciplinary collabora-
them, at the same time, learning and develop- tion amongst the fields of education and
ment practitioners, too, need to learn the lan- learning sciences, organizational development
guage spoken by business leadership. and performance improvement, computer sci-
Executives who lead the learning and develop- ence, business management, finance, and eco-
ment function in organizations and strategists, nomics is needed to support further
and those who lead departments, need to docu- development of data collection and analytic
ment organizational policy (as well as allocate methods, reporting instrumentation, and
resources for human performance and learning instructional practice. The outcomes of this
data analysts practice) and be capable of pre- call for research should aid in data-based deci-
senting responses to data-driven resource sion-making to improve organizational policy,
36 The Quarterly Review of Distance Education Vol. 17, No. 4, 2016

individualized training program recommenda- study has directly documented almost 40 refer-
tions, and organizational performance. En ences located through a systematic literature
route to these outcomes, we see vast opportu- review, along with an additional 16 references
nities for experts in each area to help educate originating from research conducted on big
organizational executives, business strategists, data and analytics in related fields of study.
and workplace learning practitioners seeking Perhaps the most notable point ascertained
to develop their skills and careers. from our review was the discovery that so little
While additional research is needed in this has been shared about recent practice improve-
area, the potential for a rapid increase in future ments and research in this area. While it may
research is limited at this time. Researchers seem counterintuitive for businesses to share
often develop studies in partnership with their practices, we know from the open source
advanced practitioners and consultants to fur- movement in software development that

S
ther develop methods and tools required to everyone benefits from improving upon shared

F
guide development of novice practitioners. source code. We also know that an economy
The current lack of knowledge and skills found built on scarcity will eventually self-implode.

O
in most practitioners working in the areas of Hence, there is much work yet to be done to
training management, talent development, and determine the full potential value of such

O
performance improvement areas increases the efforts to improve training and workplace
challenge of finding suitable partnership proj- learning initiatives. A great service would be

R 6
ects. Additionally, the opportunity to further done by researchers and practitioners who

1
develop methods and tools is limited because publish evidence in this area, documenting and

P
very few researchers are working in this rela-

0
sharing information derived from their schol-
tively new area. arship.

P 2
One area in which researchers might start
could be found in strategic business partner-

IA ©
ships, or in the case of large research institu- REFERENCES
tions with thousands of employees, in human
resource departments on campus. Addition- Adomavicius, G., & Tuzhilin, A. (2005). Toward
ally, researchers might look to collaborate with the next generation of recommender systems: A
hospitals and other large allied health organi- survey of the state-of-the-art and possible exten-
zations. Other industries with highly regulated, sions. IEEE Transactions on Knowledge and
mandatory continuing education requirements, Data Engineering, 17(6), 734–749.
a technological skilled workforce, and a large Baldassarre, L., & Finken, B. (2015). GE’s real-
time performance development. Harvard Busi-
number of employees, may also find value in
ness Review, 93(7/8). Retrieved March 3, 2016,
leveraging big data and analytics to support
from https://hbr.org/2015/08/ges-real-time-
improved training and workplace learning ini-
performance-development
tiatives. Organizations with significant opera- Berk, J. (2004). The state of learning analytics.
tions in the financial services and various T+D, 58(6), 34–39.
engineering manufacturing industries may also Berk, J. A. (2005). Identifying tangible business
be ideal partners in this endeavor. results. Chief Learning Officer, 4(3), 30–35.
Berk, J., & Magee, S. (2005). Technological consid-
erations in learning analytics. Chief Learning
CONCLUSION Officer, 4(7), 42–45.
Bersin, J. (2013). The “datafication” of learning.
This study may represent the first time a Chief Learning Officer, 12(12), 14.
review of the existing literature base on the use Bo, P., & Lillian, L. (2008). Opinion mining and
of big data and analytics in training and work- sentiment analysis. Foundations and Trends in
place learning contexts was conducted. This Information Retrieval, 2(1-2), 1–135.
Emerging Evidence on the Use of Big Data and Analytics in Workplace Learning 37

Brynjolfsson, E., Hitt, L. M., & Kim, H. H. (2011). “people data” to create value? McKinsey Quar-
Strength in numbers: How does data-driven terly, 2, 117–121.
decision making affect firm performance? George, G., Haas, M. R., & Pentland, A. (2014). Big
(Technical report). Retrieved July 25, 2016, data and management. Academy of Management
from the SSRN website: http://ssrn.com/ Journal, 57(2), 321–326.
abstract= Gandomi, A., & Haider, M. (2015). Beyond the
1819486 hype: Big data concepts, methods, and analytics.
Chen, H., Chiang, R. H., & Storey, V. C. (2012). International Journal of Information Manage-
Business intelligence and analytics: From big ment, 35(2), 137–144.
data to big impact. MIS Quarterly, 36(4), 1165– Get ready for the future of learning. (2015, June).
1188. People Management, 9, 9.
Chen, M., Mao, S., & Liu, Y. (2014). Big data: A Hagel, J. (2012). Executives turn data into internal
survey. Mobile Networks and Applications, insight. Journal of Accountancy, 213(5), 26–27.

S
19(2), 171–209. Hall, B. (2013). Will big data equal big learning?
Conertsone OnDemand Foundation. (2016). Chief Learning Officer, 12(3), 16.

F
Explore all the free learning resources. Retrieved Hartley, D. (2004). A love-hate thing. T+D, 58(6),
July 25, 2016, from http://disasterready.org/ 20.

O
courses Hartley, D. (2013). BM: Driving innovation. Chief
Chibafa, K. (2014). Why not digital? Technology as Learning Officer, 12(6), 46–47.

O
an interagency tool in the Central African Higgins, J. (2014). Bringing HR and finance
Republic. Humanitarian Practice Network, 62, together with analytics. Workforce Solutions

R 6
19–21. Review, 5(2), 11–13.

1
Daveri, F., & Mascotto, A. (2006). The IT revolu- Helios Foundation. (2014). Visibility from end to

P
tion across the United States. Review of Income end. Retrieved July 25, 2016 from http://

0
and Wealth, 52, 569–602. www.helios-foundation.org/humanitarian-

2
supply-chain/oxfam-gb-case-study/visibility-

P
David, C. C. (2006). Revving up elearning to drive
sales. Econtent, 29(2), 28–32. from-end-to-end

IA ©
Hyatt, J. (2009). The metric system. CFO, 25(7),
de Laat, M., & Schreurs, B. (2013). Visualizing
31-35.
informal professional development networks
Kettleborough, J. (2014). Big data. Training Jour-
building a case for learning analytics in the
nal, 51(3), 14.
workplace. American Behavioral Scientist,
57(10), 1421–1438. Kim, G., Lin, W. T., & Simpson, N.C. (2015). Eval-
uating the performance of US manufacturing
Dell, D. (2012). Learning analytics: How data pro-
and service operations in the presence of IT: A
vides direction. Claims, 60(10), 29–32.
Bayesian stochastic production frontier
Doan, A., Ramakrishnan, R., & Halevy, A. Y.
approach. International Journal of Production
(2011). Crowdsourcing systems on the world-
Research, 53, 5500–5523.
wide web. Communications of the ACM, 54(4),
Kristick, J. (2012). Are you learning from your
86–96.
learning programs? Chief Learning Officer,
Dolezalek, H. (2003). Measure for measure. Train- 11(8), 30–31.
ing, 40(11), 72. Laney, D. (2001). 3D data management: Con-
Dumbill, E. (2013). Making sense of big data. Big trolling data volume, velocity and variety.
Data, 1(1), 1–2. META Group Research Note, 6, 70.
Dutton, G. (2014). What’s the big deal about big Last Mile Learning. (n.d.) Last mile learning:
data? Training Magazine, 51(2), 16–19. Learning where it really matters. Retrieved July
Everson, K. (2015). Leave learning to employees. 25, 2016, from http://pilot.lastmilelearning.org
Chief Learning Officer, 14(11), 30–33. LINGOs. (n.d.). Retrieved July 25, 2016, from http:/
Freifeld, L. (2014). Go figure. Training Magazine, /lingos.org/for-anyone
51(2), 4. Manyika, J., Chui, M., Brown, B., Bughin, J.,
Gantz, J., & Reinsel, D. (2011). Extracting value Dobbs, R., Roxburgh, C., & Byers, A. H. (2011).
from chaos. IDC iview, 1142, 1–12. Big data: The next frontier for innovation, com-
Gardner, N., McGranahan, D., & Wolf, W. (2011). petition, and productivity. New York, NY:
Question for your HR chief: Are we using our McKinsey Global Institute.
38 The Quarterly Review of Distance Education Vol. 17, No. 4, 2016

Matteucci, N., O’Mahony, M., Robinson, C. & Rogers, E. M. (2003). Diffusion of innovations (5th
Zwick, T. (2005), Productivity, workplace per- ed.). New York, NY: Free Press.
formance and ICT: Industry and firm-level evi- Saunderson, R. (2014). What happens when you
dence for Europe and the US. Scottish Journal of add big data to L&D? Training, 51(2), 54–55.
Political Economy, 52, 359–386. Siadaty, M., Gašević, D., & Hatala, M. (2016).
Mayer-Schönberger, V., & Cukier, K. (2013). Big Associations between technological scaffolding
data: A revolution that will transform how we and micro-level processes of self-regulated
live, work and think. London, England: John
learning: A workplace study. Computers in
Murray.
Human Behavior, 55, 1007–1019.
Moore, C. (2005). Measuring effectiveness with
learning analytics. Chief Learning Officer, 4(5), Stoel, D. (2004, July). WWF’s corporate university:
34–39. A highly rewarding investment in human capital.
Moore, C. (2011). Diamonds in the analytics rough. TD Links Newsletter. Retrieved from https://

S
Chief Learning Officer, 10(5), 38. www.td.org/Publications/Newsletters/Links/
O’Leonard, K. (2012). Mind the global skills gap. 2004/07/WWFs-Corporate-University-a-

F
Chief Learning Officer, 11(8), 50–52. Highly-Rewarding-Investment-in-Human-
Paine, N. (2015). Game changers for learning. Capital.

O
Training Journal, 52(3), 17. Tambe, H., & Saraf, O. (2014, October). How ana-
Pape, T. (2016). Prioritising data items for business lytics helps companies improve talent ROI.

O
analytics: Framework and application to human Baseline, 1.
resources. European Journal of Operational Toal, R. (2014, August). Fundraising evaluation

R 6
Research, 249(1), 687–698. and analysis. Retrieved from https://www

1
Pease, G., & Beresford, B. (2013, June). Big data .fundsforngos.org/featured-articles/

P
analysis drives not-for-profit performance. T+D,
fundraising-evaluation-analysis/

0
67(6), 22–24.
Toterhi, T. (2014). Make sure big data adds up.
Petticrew, M., & Roberts, H. (2008). Systematic

P 2
Training Magazine, 51(4), 14.
reviews in the social sciences: A practical guide.
Tozman, R. (2012). New learning analytics for a

IA ©
Oxford, England: Blackwell-Wiley.
Roberts, B. (2007). Data-driven human capital deci- new workplace. T+D, 66(2), 44–47.
sions. HR Magazine, 52(3), 105–108. Young, K. (2015, October). Intelligent analytics.
Roberts, P. (2015). The CFO and CHRO guide to Training Journal, 56–59.
employee attrition. Workforce Solutions Review, Wroe, N. (2012). Innovations in talent analytics.
6(1), 8–10. T+D, 66(8), 30–31.
AUTHOR BIOGRAPHICAL DATA

Ray J. Amirault (Lead Editor) holds a BS in Florida State University where she teaches
computer science from The University of West courses on instructional design and research

S
Florida, and both an MSc and PhD in instruc- methods for new and emerging technologies.
tional systems from Florida State University. Her research investigates social knowledge

F
Ray has 20 years of experience in instructional activities in online environments, looking at

O
design and e-learning, and has taught and issues such as identity and network develop-
designed courses and/or programs at The Flor- ment, knowledge brokering and sharing, and

O
ida State University, Florida Atlantic Univer- privacy and ethical concerns. The computer-
sity, Wayne State University, and Illinois State mediated environments in which her research

R 6
University. Ray’s publications examine tech- has been situated include both formal and

1
nology’s impact on education, often within the informal learning, with technologies ranging

P
historical timeline of the university movement. from learning management systems to social

0
Ray serves on the editorial board of three media tools. She also serves as co-editor-in-

P 2
instructional technology journals, is a perma- chief of The Internet and Higher Education.
nent editor for the International Section of the

IA ©
Quarterly Review of Distance Education, and Lisa A. Giacumo is an assistant professor of
has served as a solicitation reviewer for the organizational performance and workplace
National Science Foundation. He is currently learning in the College of Engineering at Boise
an instructional technologist at Illinois State State University. Her current research interests
University. Ray is a native of suburban Chi- are at the intersection of instructional design
cago and is a photography enthusiast. and workplace learning, including: data-based
performance improvement solutions, social
Jeroen Breman is the learning and develop- and digital media in training and workplace
ment consultant for Grid U, the corporate uni- learning, global training initiatives, and com-
versity of Northwest Lineman College, at their petency development in instructional design-
Meridian, ID, campus. His current research ers.
interests focus organizational performance and
workplace learning, instructional design, and Mary O’Brian holds a BS in special educa-
blended learning solutions in remote learning tion, an MS in psychology, and a doctorate
environments. from Illinois State University. Mary has
worked for over 30 years as a special educator,
Vanessa Dennen ([email protected]) is a pro- a school psychologist, school administrator,
fessor and program coordinator of instruc- and a faculty member in various educational
tional systems and learning technologies at settings. She has completed programs for both

The Quarterly Review of Distance Education, Volume 17(4), 2016, pp. 75–76 ISSN 1528-3518
Copyright © 2016 Information Age Publishing, Inc. All rights of reproduction in any form reserved.
Reproduced with permission of the copyright owner. Further reproduction prohibited without
permission.