Advanced Engineering Informatics 52 (2022) 101508

Contents lists available at ScienceDirect

Advanced Engineering Informatics

journal homepage: www.elsevier.com/locate/aei

An AI-based open recommender system for personalized labor market driven

education
Mohammadreza Tavakoli a , Abdolali Faraji a , Jarno Vrolijk b , Mohammadreza Molavi c ,
Stefan T. Mol b ,∗, Gábor Kismihók a
a
Leibniz Information Centre for Science and Technology (TIB), Germany
b University of Amsterdam, The Netherlands
c Amirkabir University of Technology, Iran

ARTICLE INFO ABSTRACT

Keywords: Attaining those skills that match labor market demand is getting increasingly complicated, not in the last place
Recommender systems in engineering education, as prerequisite knowledge, skills, and abilities are evolving dynamically through
Open educational resources an uncontrollable and seemingly unpredictable process. Anticipating and addressing such dynamism is a
Educational data mining
fundamental challenge to twenty-first century education. The burgeoning availability of data, not only on
the demand side but also on the supply side (in the form of open educational resources) coupled with smart
technologies, may provide a fertile ground for addressing this challenge. In this paper, we propose a novel,
Artificial Intelligence (AI) driven approach to the development of an open, personalized, and labor market
oriented learning recommender system, called eDoer. We discuss the complete system development cycle
starting with a systematic user requirements gathering, and followed by system design, implementation, and
validation. Our recommender prototype (1) derives the skill requirements for particular occupations through an
analysis of online job vacancy announcements; (2) decomposes skills into learning topics; (3) collects a variety
of open online educational resources that address those topics; (4) checks the quality of those resources and
topic relevance with three intelligent prediction models; (5) helps learners to set their learning goals towards
their desired job-related skills; (6) recommends personalized learning pathways and learning content based on
individual learning goals; and (7) provides assessment services for learners to monitor their progress towards
their desired learning objectives. Accordingly, we created a learning dashboard focusing on three Data Science
related jobs and conducted an initial validation of eDoer through a randomized experiment. Controlling for the
effects of prior knowledge as assessed by means of a pretest, the randomized experiment provided tentative
support for the hypothesis that learners who engaged with personal recommendations provided by eDoer to
acquire knowledge of basic statistics, attained higher scores on the posttest than those who did not. The
hypothesis that learners who received personalized content in terms of format, length, level of detail, and
content type, would achieve higher scores than those receiving non-personalized content was not supported.

cost effective, scalable, and sustainable means to match the exponen-

tially growing array of open educational resources to (the needs of)
‘‘Ensure Inclusive and Equitable Quality Education and Promote Life- individual learners, regardless of socio-economic status and/or demo-
long Learning Opportunities for All’’ - United Nations Sustainable
graphic background. Indeed, where top-quality educational resources
Development Goal 4 [1].
were once solely accessible to the privileged few who were enrolled
in top-tier educational institutions (mostly) in developed nations, the
1. Introduction
growing trend of opening up such resources, together with technologi-
cal developments that allow for the matching of content to learners on
With the clock ticking on the United Nations’ Sustainable Develop-
a massive scale, has created opportunities to distribute and disseminate
ment Goal pertaining to quality education, the time is ripe to develop

∗ Corresponding author.
E-mail addresses: [email protected] (M. Tavakoli), [email protected] (A. Faraji), [email protected] (J. Vrolijk), [email protected] (M. Molavi),
[email protected] (S.T. Mol), [email protected] (G. Kismihók).

https://doi.org/10.1016/j.aei.2021.101508
Received 30 April 2021; Received in revised form 10 November 2021; Accepted 19 December 2021
Available online 24 February 2022
1474-0346/© 2022 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

such educational resources more equitably, inclusively, and effectively, and finally (5) assess (changes in) prerequisite knowledge for requisite
to all those who seek them. skills.
Although altruism and the ‘‘feel-good factor’’ have been identified A number of occupational taxonomies (e.g., ESCO and O*NET)
as some of the main drivers of the movement to open up educational exist that may be leveraged to provide information about occupations.
resources [2], to date the word open has remained more of a legal However, most of these taxonomies are updated through a largely
designation, than a harnessed potential. There are benefits associated manual process, meaning they are time-consuming and expensive to
with tapping into the vast array of Open Educational Resources (OER) construct, and also susceptible to being outdated [16]. Alternatively,
that go beyond just making them accessible to people who may other- text-based algorithms can be developed to extract those topics that
wise not be able to access education. First, in light of the burgeoning
are on the one hand manifest in corpora of job vacancies, and on the
amount of publicly available textual data [3], there are opportunities
other, covered by existing educational materials in an effort to help
for more explicitly mapping educational content to the demands of the
learners to build their learning path [13–15,17]. To offer personalized
labor market, therewith enhancing learners’ motivation, learning effec-
educational services (e.g recommendation and search services), we thus
tiveness, and employability. Indeed, to date, efforts at personalizing
need to (1) extract properties of educational materials (e.g., quality and
educational content to learners is often backward-looking (i.e., where
learners came from) as opposed to forward-looking (where they are metadata [18,19]), (2) detect preferences of learners (e.g., preferences
going). Second, greater and greater demands are being placed on teach- regarding format and the time investment associated with mastering
ers, not only in terms of the ICT (Information and Communications the educational content [4,12]), and (3) match between the resources
Technology) heavy teaching methods they need to master, but also and the learners [20,21]. However, previous efforts to build such edu-
in terms of increasing student numbers and courses they may have to cational systems by drawing on the rapidly growing amount of online
teach. As we shall illustrate later, the ability, on the part of students, to and open educational resources [22,23] revealed the lack of high-
automatically identify and be recommended OERs based on where they quality OER metadata, and quality control [18]. These issues seriously
stand and where they are going may complement traditional courses curtail the accessibility of OERs, an issue that may be tackled through
and may ultimately serve to make teachers’ workloads more man- the deployment of high-quality OER search and recommendation ser-
ageable. Third, many educational curricula crush student self-directed vices [20,21]. Our aim, therefore, besides contributing to the resolution
learning, proactivity, sense of control, and autonomy by dictating what of OER metadata quality issues by leveraging the work of [12], is
is to be learned and when it is to be learned, without providing learners to generate metadata (e.g., technical quality of video/sound/text, and
with a sense of the bigger picture, or why they are learning what how relevant the content is to a target topic) by having learners interact
they are having to learn. The information asymmetry that this entails, with our OER recommendation system (i.e, through point-of-experience
means that all too often students are just passive receivers of education, surveys that solicit quality feedback).
as opposed to them taking guided decisions and expending motivated In this paper, we set out to address the aforementioned challenges
effort towards shaping their own future. It is against this backdrop that
and present a software prototype (with an initial focus on data sci-
we started working on designing and constructing a vehicle that can
ence related jobs) to provide personalized, open educational content
connect learners to the educational contents that they seek and/or need
recommendation in an effort to help learners master their target skills
regardless of their geographic location, demographic characteristics,
by:
and/or formal educational qualifications.
Recent decades have seen educational environments changing dra- • Defining those topics that need to be learned to acquire a skill
matically in response to the increasing demand for online personalized • Empowering learners to construct their own learning trajectories
learning [4]. There is a growing need for online personalized educa- based on labor market information and OERs
tional services because of (1) the rapid evolution in both the quantity • Recommending personalized educational resources based upon
and quality of skills demand [5–7]; (2) the gap between knowledge our automatic quality control models
(and skills) that job markets require and the training that formal edu- • Providing a two-layer assessment that evaluates the level of the
cational programs offer [8–11]; and (3) the global challenges for work
learner both with regard to target skills and topics.
and education due to the emergence of the COVID-19 pandemic [12].
Consequently, we are facing exponential growth in educational We end this article by describing our effort to validate our prototype
resources (such as Online and Open Educational Resources) that are by conducting a randomized experiment in which 175 paid participants
being produced and disseminated on an unprecedented scale, and were asked to learn about Basic statistics for engineers. The specific ob-
published in different contexts (e.g., location, language, discipline, jectives of this study were (i) to examine whether learners who engaged
expertise level, and format) [13,14]. However, the heterogeneity and with our system acquired more knowledge than learners who did not,
(lack of) targeted distribution of these educational contents leads to and (ii) to examine, among those users who engaged with the system,
a number of problems for learners that limit their usefulness. First, whether learners who received personalized content recommendations
learners may not understand which components they need to learn acquired more knowledge than those who did not.
to fulfill skill (or knowledge) requirements [15] let alone the skills
This paper is organized as follows: Section 2 depicts the state-of-
demanded by the labor market. Second, even if they knew what it was
the-art regarding the different components of our proposed system,
that they were seeking to learn, learners are unlikely to be able to
followed by Section 3, in which we explain the process of construct-
distinguish between high quality and low quality educational resources.
In sum, confronted with an abundance of learning materials, learners ing our proposed system including the requirement gathering/analysis
may be overwhelmed and will unlikely be able to plot and follow their step, matching jobs with their required skills, decomposing skills into
own effective learning path without directional guidance in the form meaningful educational components (i.e., topics), collection and quality
of personalized educational recommendations. control of educational resources, building our recommendation system,
According to the state of the art and our requirement gathering, and implementing our proposed learning dashboard. Section 4 presents
there is a need for intelligent systems that help learners to (1) be up- our hypotheses, and discusses the methods and outcomes of the valida-
to-date about the required skills for their target (current or future) jobs; tion study into our prototype. Subsequently, in Section 5 we summarize
(2) be informed about the components (topics) that they need to learn the main findings pertaining to to our research questions, objectives,
for each skill; (3) build their own learning path towards the required and hypotheses before discussing the implications, limitations, and
skills; (4) find personalized learning materials according to their path; future directions. In Section 6 we conclude the paper.

2
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

2. Related work 2.4. Content recommendation for further education

Due to the dramatic evolution of skill requirements in recent

2.1. Automatic extraction of online educational content properties
decades, matching processes between skill demand and supply on the
labor market are getting more and more complicated as skills dynami-
In order to build their own learning trajectory [17], it is imperative cally change through an uncontrollable and seemingly unpredictable
for learners to have insight into the specific topics and properties process [6,7]. Employees’ staying abreast of these changing labor
that are addressed by particular Online Educational Resources. To market demands, and acquiring the concomitant skills is a prerequisite
tackle this issue, some studies have developed semantic [13,14,24] for sustainable employability [6,35]. However, when it comes to the
and machine learning based [15,25,26] methods to extract topics and offerings of Educational institutions these often lag behind, increasingly
other pertinent metadata from educational resources. For instance, resulting in a gap between the skills (and associated learning content)
summarized content, derived by extracting keywords from educational that educational institutions impart in their students, and the current
resources [27], may be linked to other relevant contents such as skills that job markets demand [9,11,36]. To mitigate this mismatch,
Wikipedia [28]. Other studies have focused on extracting the type of ed- understanding labor markets dynamics and information is critical [4].
On the one hand, this understanding requires the decomposition of
ucational content. For example, [29] applied deep neural architectures
jobs into skills and other relevant building blocks. Recent efforts in
to extract the video types (e.g., PowerPoint presentation, code writing,
this area have applied text mining/machine learning techniques [3,37]
or the instructor talking to the learners) from educational videos.
and semantic-based [38] methods to text data obtained from online
Although further work is sorely needed in this area [15], these stud-
vacancies. On the other hand, the extracted skills from the demand
ies illustrate the viability of automatically extracting properties from
side must go through a matching process, by collecting related high-
educational resources that may then be used as a basis for matching quality educational resources which may address the learners’ needs
and recommendation. towards achieving those skills, and using recommendation methods
(e.g., Content-based and Collaborative filtering) to provide the most
relevant content for learners to help them acquire those skills effi-
2.2. Quality evaluation of online educational content
ciently [4,12]. According to the extant literature, the area of labor
market based education requires more research and development to
The sheer amount and heterogeneity of online and open educational facilitate the provision of more personalized and up-to-date services to
contents that are created and uploaded to the internet on a daily individual learners.
basis, places limitations on that which can be achieved with manual
quality control. This is problematic, because high-quality metadata 2.5. Research questions and objectives
are a mandatory prerequisite of any data driven (not only educa-
tional) service [30]. In light of this, some studies have attempted to Based on the state of the art, it is clear that (1) online educational
define metrics (e.g., completeness and consistency of metadata [31], services that offer open and online educational resources to learners
provenance, and accuracy of metadata [32]) to assess OER metadata may address the pressing need for inclusive, equitable, and effective
quality. [33], based on existing developments in quality models in online education, while at the same time becoming ever more viable
as technology is developing. Therefore (2) the time is ripe to focus
e-learning, semantic-based methods, and NLP techniques, developed
greater research attention on personalized high-quality educational
a quality assessment framework. Building on this line of work, [18]
resource recommendations. (3) This is especially important for learn-
developed a metadata scoring method, which can be applied to the
ers in further education, who need help to build their own learning
definition of a model to predict OER quality. Also, [19] proposed a
trajectories towards their desired jobs, thus (4) offering learners those
quality evaluation model for open educational videos: their quality
existing educational contents that they need to learn in order to acquire
was effectively predicted on the basis of video transcripts, popularity the skills which are demanded by the labor market, can potentially
metrics (i.e., ratings, likes, dislikes), and length. Despite the progress in enhance learners’ autonomy, and motivation, thereby enhancing the
this area, the fast development of disparate online and open educational (effectiveness of the) individual learning experience. In light of these
resources calls for researchers to continue to improve the methods that points, the main research questions of this paper are:
underpin quality evaluation models [18].
• RQ1. What are the key learner requirements (e.g., vis-a-vis type of
content, level of detail, and assessment) for labor market driven
2.3. Recommending open online educational content personalized education?
• RQ2. How may learners be aided in constructing their own learn-
The literature on high-quality OER recommender systems is very ing trajectories based on labor market information and open/
limited [21]. However, in recent years, some studies have advanced online educational resources?
• RQ3. What models provide adequate and accurate predictions
recommendation algorithms based on ontologies, linked data, and
about the quality of online educational content?
open-source Resource Development Framework (RDF) data to leverage
• RQ4. How can we combine job/skill decomposition, quality pre-
semantic content [21,23]. For example, [22] defined an ontology for
diction of educational resources, and feedback to learners to
learners, learning objects, and their environments to provide adaptive
develop a personalized open educational content recommender
recommendations, based on similarities between object properties. In
system for learners in further education?
a different vein, [20], using existing ontologies, examined the Cold
Start problem [34] in the realm of micro OERs, by defining rules, based 3. Method
on recommended sequences of learning objects. Finally, [4,12] built
an OER recommendation system to help learners achieve skill-based 3.1. Requirement analysis
learning objectives using (1) a text mining approach to extract skills
from online job vacancies, and (2) a gradient descent algorithm to pre- First, we collected relevant stakeholder requirements to further
dict user preferences based on their ratings of previously recommended define our objectives and guide our investigation. For this exercise,
educational resources. we built an initial and bare-bones OER recommender prototype so as

3
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

to be able to showcase our approach to key stakeholders. Through 3.1.4. Lessons learned
qualitative interviews, this prototype was evaluated by 23 subject By analyzing participants’ ratings regarding these potential user
matter experts (e.g., university instructors and Ph.D. students) with requirements, we prioritized and constructed the following services for
significant experience in both industry and learning/teaching [4,12]. learners:
Based on their feedback, we designed a questionnaire1 to cap-
ture the needs of those stakeholder groups that we expected to be • Service_1: Personalized Search. 𝑅𝑒𝑞1 and 𝑅𝑒𝑞2 ( Table 1),
potentially important beneficiaries of our learning recommender sys- clearly received the highest ratings among all requirements. There-
tem. We identified the following stakeholder groups (personas)2 : fore implementing an educational resources search service, which
provides accurate and high-quality search results to address indi-
• Group1. Recipients (e.g., Learners, Researchers, Students) vidual learning needs, became one of our top priorities. Clearly,
• Group2. Deliverers (e.g., Professors, Lecturers, Study Counselors) the personalization and the content-quality of the results of such
• Group3. Facilitators (e.g., Managers, Educational Support Staffs) a service are critical as demonstrated by 3.1.2, where learners
pointed to the lack of personalization and problems in identifying
We obtained 13 potential user requirements from the initial quali-
high-quality learning content as two of the most important barriers
tative interviews, which we then presented to survey participants (see
to using open/free educational resources. Hence we focused on
Table 1), asking the latter to rate those in terms of their importance
the context of the learners (e.g., job, skill-set, expertise level,
and frequency of use. Since in this study we focus solely on the learner
language), and their learning preferences (e.g., their preferred
perspective, the following subsections showcase the most important
format (e.g video or web pages)).
outcomes and lessons learned from Group1 members.
• Service_2: Goal-driven Learning Content Recommendations.
According to 𝑅𝑒𝑞2, 𝑅𝑒𝑞3, and 𝑅𝑒𝑞5, learners desire a service that
3.1.1. Personal information
helps them (1) explicate their learning objectives, (2) find suitable
Altogether 47 learning recipients (Group 1) from 10 countries com-
learning pathways that fit to their context (preferences), and (3)
pleted our questionnaire and returned usable data. Of these Group1
receive the most relevant and highest-quality learning resources
participants, 43.2% were female, 51.3% were male, and 5.5% did not
needed to meet their learning objectives.
provide any information on their gender. Of the participants, 12.8%
• Service_3: Elucidating Job Skill Requirements. Based on 𝑅𝑒𝑞4
had completed High-school or lower, 14.9% had a Bachelor, 36.2% had
and 𝑅𝑒𝑞6, the need can be observed to match jobs and the skills
a Master, 34% had a Ph.D., and 2.1% had completed other educational
that are required to be effective in those jobs. This should be
degrees or qualifications.
accompanied by visualization, which helps inform users about
those skills they need to acquire. Based on this information one
3.1.2. Current skill progression towards desired occupations
can set learning targets and obtain (and ultimately learn) relevant
Survey participants’ reported informing themselves about skill de-
learning content.
mands in the following ways: 86.5% while performing their every-
day tasks, 62.2% through reading related papers or news, 54.1% by • Service_4: Learning Progress Monitoring. Learners also ex-
inquiring with their supervisors, and 40.5% through job vacancy an- pressed a strong interest in monitoring their progress towards
nouncements of positions they apply to. Moreover, they mentioned their learning goals (𝑅𝑒𝑞8). Accordingly, we found it essential
courses (83.8%), educational videos (78.4%), books (72.9%), and Web to provide an assessment service, which would help users to test
pages/documents (64.9%) as dominant resources they used to de- the knowledge they set out to acquire. Additionally, we decided
velop themselves towards skills required by employers. Finally, with to provide further insights (through numbers, charts, etc.) about
respect to open learning content for their self-development, partic- users’ progress towards each of their learning goals.
ipants bemoaned (1) the lack of personalization, (2) the identifica-
tion/localization of high-quality learning content, and (3) the time- 3.2. Labor Market Intelligence
consuming search process, as the most pressing problems.

In order to match jobs to their skill requirements (𝑅𝑒𝑞4 and 𝑅𝑒𝑞6),

3.1.3. Importance and frequency of use of the potential requirements
we deployed a Labor Market Intelligence (LMI) component to capture
Participants rated the importance (1: Not at all important - 5: Very
up-to-date skill requirements for jobs relevant to this study.
important), and frequency (1: Never - 5: Daily) of usage for each poten-
tial user requirement. Once data collection was complete, we calculated In an initial effort demonstrate the applicability of our system,
the average of their ratings for each of the requirements and normalized we decided to focus on Data Science related jobs. We did so because
the average rates using Min–Max Normalization as (1) in which we these jobs are both in high demand and particularly prone to change.
replaced the values with the average rates. Table 1 shows the potential We selected three associated jobs: Data Scientist, Data Analyst, and
user requirements, normalized average importance ratings, normalized Business Analyst. Subsequently, we used a sample data-set of English
average frequency ratings, and the composite rate (multiplication of the job vacancies from Monster.com,3 which included 21,937 vacancies and
normalized importance and frequency rates) which have been sorted their related skills.
based on the composite rates. Subsequently, we calculated the rate of occurrence for each of the
𝑉 𝑎𝑙𝑢𝑒 − 𝑀𝑖𝑛𝑖𝑚𝑢𝑚𝑣𝑎𝑙𝑢𝑒𝑠 skills in the target jobs and set the importance of the skills in each job
𝑁𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑_𝑣𝑎𝑙𝑢𝑒 = (1) based on this occurrence rate. We used this importance rate to sort the
𝑀𝑎𝑥𝑖𝑚𝑢𝑚𝑣𝑎𝑙𝑢𝑒𝑠 − 𝑀𝑖𝑛𝑖𝑚𝑢𝑚𝑣𝑎𝑙𝑢𝑒𝑠
skills that learners need to learn. Based on this process, the following
six skills were selected to represent our target jobs as they achieved the
highest importance rates across our target jobs: Python programming,
1
The questionnaire is available on: R programming, Statistics, Machine learning, Data Visualization, and Text
https://tib.eu/umfragen/index.php/survey/index/sid/977178/newtest/Y/ mining.
lang/en.
2
It should be mentioned that we allowed participants to answer our
questionnaire from the perspective of multiple personae. This was important,
3
as a single person can fulfill different roles in a learning process (e.g., a person The data-set is available on: https://www.kaggle.com/PromptCloudHQ/
can be a lecturer and manager at the same time). us-jobs-on-monstercom/version/1.

4
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

Table 1
Average importance and frequency ratings for potential user requirements.
Requirement Importance rate Frequency rate Composite rate
[0–1] [0–1] [0–1]
𝑅𝑒𝑞1. Finding learning content about a problem I am working on at the moment 1.00 1.00 1.00
𝑅𝑒𝑞2. Identifying high-quality content which fulfills my learning needs 0.81 0.68 0.55
𝑅𝑒𝑞3. Knowing where to start learning when I need a new skill for my studies/job 0.75 0.38 0.29
𝑅𝑒𝑞4. Identifying which skills are required for my current/future job 0.70 0.36 0.25
𝑅𝑒𝑞5. Defining my own goals towards jobs I find attractive 0.53 0.29 0.15
𝑅𝑒𝑞6. Identifying which skills are required for my degree 0.40 0.27 0.11
𝑅𝑒𝑞7. Finding out how I can improve my skillset in order to qualify for my desired job 0.58 0.18 0.10
𝑅𝑒𝑞8. Monitoring my learning progress towards desired skills 0.23 0.24 0.06
𝑅𝑒𝑞9. Making sure that my learning objectives meet job requirements 0.40 0.11 0.04
𝑅𝑒𝑞10. Identifying which skills are the most important ones in terms of contributing to expected salary 0.05 0.07 0.004
𝑅𝑒𝑞11. Visualizing potential skill targets 0.05 0.05 0.003
𝑅𝑒𝑞12. Identifying which jobs I can fulfill with my skillset 0.15 0.00 0.00
𝑅𝑒𝑞13. Visualizing the structure of the content that I need to master to achieve my skill targets 0.00 0.04 0.00

Table 2 and open educational resources and the fields we needed to apply our
Collected resources for each skill. automatic models6 :
Skills Number of Number of covered Number of
collected playlists educational videos topics • Source. Records the original location of the content.
Python programming 8 502 26 • Format. The format (e.g., Video, Web page, or Book chapter) of
R programming 4 185 12 the content. This was set based on the source and file extension
Statistics 9 621 27 of the resources. For example, this field was set to Video for the
Machine learning 9 472 35
resources from Youtube.
Data visualizing 8 257 14
Text mining 6 194 18 • Title. Records the title of the content.
• Description. Records the description of the content.
• Transcript. Records the transcription of the content. This field was
set based on the transcript of the videos, and the content of the
3.3. Educational topic detection for selected skills web pages, and book chapters.
• Rating. User ratings of the content. This field was calculated dif-
In order to recommend open learning content for the selected skills ferently (e.g., based on 5 point rating scales or likes and dislikes)
(𝑅𝑒𝑞2 and 𝑅𝑒𝑞3), we needed to decompose each skill into meaningful in the different sources. Therefore, we normalized the ratings for
learning Topics. Therefore, we extracted learning topics for these six each of the resources.
skills by applying Latent Dirichlet Allocation (LDA [39]) to the transcripts • Length. This field shows the content length (in seconds only for
of existing educational materials. Specifically, we used the method videos).
proposed by [15] to extract learning topics and determine the degree to • View Count. Total number of times that the educational content
which those topics were reflected in each educational resource. Finally, had been viewed by users.
we asked three experts to prioritize each of the extracted topics with
3.4.2. Filtering based on quality and relevance
an eye on skill development. Table 2 shows the number of collected
To provide high-quality educational content, which was one of the
playlists (each of which comprises the educational resources per skill),
key outputs of our requirement analysis step (Req2), we applied the
the number of covered educational videos, and the final number of
following filtering procedure on the collected OERs and other available
extracted topics for each skill. It should be mentioned that some of the educational resources:
topics were part of more than one skill (e.g., Linear Regression was a
topic of both Machine Learning and Statistics skills) • Topic-based filtering. In order to remove educational content
that did not fit the search keywords detailed in the previous
section, we used the output of our topic models that was described
3.4. Incorporation of educational content
in Section 3.3. Specifically, we extracted the target topic of each
educational resource using our topic models, and removed those
In this section, we describe how relevant high-quality open educa- resources for which the extracted target topic did not match its
tional resources were collected, filtered, and labeled (𝑅𝑒𝑞1 and 𝑅𝑒𝑞2). search keywords. For instance, if a video was the result of the
We also depict how assessments were connected to the final set of search keywords ‘‘Machine Learning Linear Regression’’, but our
educational resources included in our recommender (𝑅𝑒𝑞8). model detected its focus as ‘‘Support Vector Machine’’, we re-
moved it from our resource list. This step resulted in the removal
3.4.1. Collection of online educational resources of a total of 1,116 resources (906 of which were video- and 210
To collect open educational content for the six skills and their topics, of which were textual resources)
we performed a search on Google and Youtube4 using the concatenation • Metadata-based filtering. Previously, [18,40] showed that the
metadata quality of OERs is indicative of their content quality.
of the skill and the topic (e.g., ‘‘Python programming Conditions’’)
Based on this finding, we created a binary classifier to sort
as the search keywords. We collected 3,228 educational resources5
educational resources into a high-quality and a low-quality group.
which included 2,514 educational videos and 724 text-based resources
By applying their machine learning model, educational resources
(e.g., web pages, lecture notes, and book chapters). For each resource,
with a predicted low-quality content (a total of 727 resources
we collected the following fields based on the available fields for online
of which 621 were video-and 106 were textual resources) were
removed from our educational content collection.

4
Using Pafy python-youtube library: https://pypi.org/project/pafy/.
5 6
This is a new data-set and is different from the one we used for the topic It should be mentioned that some resources in our data-set did not include
detection step. all the mentioned fields.

5
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

Table 3 • Is a Classroom-Based Instruction. This field is a Boolean value that

Number of resources which passed through our filtering steps.
captures whether the resource has been recorded as a university
Skills Number of Avg number of class or not.
educational resources resources per topic
Python programming 124 4.77
R programming 49 4.08 3.4.4. Implementation of learning progress monitoring
Statistics 209 7.74 To produce well-defined and relevant assessments, three experts
Machine learning 263 7.51
generated and carefully reviewed multiple-choice questions (test items)
Data visualizing 100 7.14
Text mining 120 6.67 for each topic. In this process, a question was selected to be added to
our test items, when all reviewers found it appropriate to assess the
knowledge of learners in the topic(s) that the question targeted. This
resulted in a repository of topic-based and skill-based test items. In our
• Quality-based filtering. In our last filtering step, we checked prototype, we implemented two different types of assessment, each of
whether OERs and other available educational resources fit the them are generated dynamically, according to the individual progress
description of the target learning goal of the content (based of each learner:
on the Wikipedia page of the search keyword we used to col-
lect the content), and the level of prior learners’ satisfaction in • A progress assessment is a test which only contains test items
terms of content ratings and view counts. This was accomplished related to topics. This test validates the progress of a user, when
through the quality prediction model proposed by [19]. This they transit between consequent topics within a skill. Learners
model leverages the similarity between the transcription of edu- can only start a new topic if they pass the assigned progress
cational resources and the description of their target topics (from test of the prerequisite topic(s). In case a learner passes a topic
Wikipedia) in addition to their popularity features (e.g., rating associated with a target skill, the topic is marked as completed in
and view count) to determine quality. To apply the model on all corresponding (related) skills listed in our recommender. For
our data-set, we rebuilt their proposed prediction model based instance, if a learner passes the topic ‘‘Linear Regression Concept’’
on the features that existed in all of our collected resources when studying for ‘‘Machine Learning’’ skill, this topic will also
(i.e., Transcript, Rating, and View Count) which led to 79.2% of be completed for the skill ‘‘Statistics’’, even though this skill is
the F1-score on their published data-set. As a result of this step, a not among the skill targets of the learner. This method helps
total of 631 (547 video- and 84 textual) resources were removed track individual development, by monitoring knowledge and skill
from our collection. proficiency levels across topics and skills, within and beyond
Through the application of the aforementioned filters, we distilled individual learning objectives.
764 high-quality (440 video- and 324 textual) OERs and other available • A skill assessment can be interpreted as an assessment of skills
educational resources, covering all topics 3.3 in our six target skills (see (i.e., a topic aggregate), and can be used to provide feedback
Table 3). It should be noted that the number of educational resources to learners about applying and combining acquired knowledge
for each topic ranged between 3 and 10, and that we had at least one (topics) areas in relation to a specific skill. Therefore, these
video and one textual resource for each. Moreover, in our data-set, assessments include questions that cover all topics associated with
there were resources that addressed more than one topic (e.g., an edu- a specific skill. Learners complete these assessments as soon as
cational video could cover both Linear Regression and Gradient Descent ). they have mastered the different components (topics) of a target
skill.

3.4.3. Educational resource labeling Using progress and skill assessments, a learner can continuously
To generate the personalized recommendations for the learners, evaluate their level of knowledge in a fine-, and coarse-grain manner.
we analyzed and labeled all of the educational resources that were
retained. Some features such as Source, Format, Transcript, Rating, and 3.5. Personalized open learning content recommendation
View Count had already been extracted automatically (see 3.4.1). Ad-
ditionally, for each skill, we asked two experts to review and label the In this section, we demonstrate our proposed personalized recom-
resources (see below). As a result, the following features were collected mendation system for learners to address 𝑅𝑒𝑞1 and 𝑅𝑒𝑞2.
for all filtered educational resources:
3.5.1. Learner profile
• Length. As we extracted the length of educational videos (in
Based on the features we collected for educational resources 3.4,
seconds), we asked experts to estimate how long it would take
we also defined features for the preferences of each and every learner.
learners (in seconds) to scrutinize the text-based educational re-
These features are described in Table 4. Based on possible feature
sources. Afterward, we grouped educational resources in such a
values, we created a long-, and a short-term 15-dimensional prefer-
way that we had groups with a similar number of resources, that
ence vectors for each learner which included the following features:
we could describe to the learners easily. Therefore, we created
3 groups of Short < 10 min (included 308 resources), 10 min < Length-Short, Length-Medium, Length-Long, Detail-Low, Detail-Medium,
Medium < 20 min (included 225 resources), and Long > 20 min Detail-High, Strategy-Theory, Strategy-Example, Strategy-Both, Class-based,
(included 231 resources) resources. Non-class-based, Content-Video, Content-Book Chapter, Content-Web Page,
• Level of Detail. This feature captures the level of detail in which Content-Slide. Each feature value in a vector shows how much (a float
a specific content addresses a target topic.7 Experts assigned the value from 0 - the lowest, to 1 - the highest) a learner prefers receiving
following labels to the resources: Low Detail, Medium Detail, High learning resources with that feature. The long-term vector is used as
Detail. the basis for our learning content recommendation. Therefore, the
• Learning Strategy. We defined three learning strategies of Theory- complete history of each learner’s feedback (5-scale ratings for the
based, Example-based, and Mixed (which includes both theory and recommended educational contents) until the recent updating period
example) based on [41], and asked experts to label resources is taken into account. The short-term vector shows learners’ feedback
accordingly. in the recent updating period (last one month) and it affects the
long-term vector at the end of each updating period; therefore, the
short-term vector is emptied at the starting point of each updating
7
The topic can be a concept, formula, or an API. period and updated after each feedback from the learner. The long-term

6
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

Table 4
Preference features.
Feature Possible values Notes
Length Short, Medium, or Long Learner’s preference about the length of educational resources
Detail Low, Medium, or High Learner’s preference about the level of details in educational resources
Learning Strategy Theory-only, Example-only, or Both Learner’s theoretical knowledge orientation
Classroom-based Yes or No Learner’s preference about learning content originated from classrooms
Content Format Video, Book, Web page, Slide Learner’s preference about learning content formats

vector helped us to capture the learners’ preferences while using the 3.6.1. Registration and goal setting
recommended resources (it should be noted that the long-term vector The registration path consists of three consecutive steps, each serv-
is configured to place more weight on the recent ratings). We defined ing a different purpose: (1) In the first step we collect the necessary
the updating period as a configurable period value (which could be set demographic information from new learners, including their name,
in our system), and set it to one month in this version of our prototype. email address, gender, and geographical location (country and city).10
When a learner registers in our system for the first time, we ask (2) In the second step, learners search for and select a target job.
questions regarding all preference features in order to populate the
Subsequently (as an implementation of Service_3 depicted in 3.1.4), we
long-term preference vector. This is done by transforming the selected
show the required skills for the selected job by using our labor Market
values into to the corresponding values (float number between 0 and
Intelligence 3.2, and ask learners to select those skills they want to
(1) in our preference vectors. For instance, when a learner prefers Long
master. In addition, users can search and select complementary skills
content, the Length-Long feature is set to 1, while the Length-Short and
(not connected to their target job) and add them to their target skills
Length-Medium features are set to 0. As another example, if a learner,
selects 3 on a 5 point rating scale rating regarding the video contents, manually.11 (3) The third (and last) step consists of setting learning
the Content-Video feature is set to 0.5. preferences by answering a number of questions (see Section 3.5.1), to
When the learners complete a learning content, we consider their further calibrate the learning content recommender algorithm for each
feedback, which is a 5-scale rating, to update their short-term profile. particular learner.12
For instance, assume that after recommending two pieces of learning
content with a High level of detail to a learner, and we receive the
3.6.2. Personalized learning
following feedback ratings: (1) 3 in a 5-scale rating (which means 0.5
To provide Service_2 (see 3.1.4), a curriculum page was designed to
out of 1 in our system), and (2) 5 out of 5 (which means 1 out of 1).
As a consequence the Detail-High feature of the short-term vector is set structure and monitor the advancement of learners with respect to their
to 0.75 (which means 4 in a 5-scale rating) for the learner. target skills and related topics. Learners can visualize their personalized
At the end of each updating period (which was set to one month), curriculum by selecting a skill. Once the skill is selected, the related
we updated the long-term vector by calculating the average of the list of topics are displayed, sorted by their priority (see 3.3).13 Each
current long-term vector and the short term vector. This updating pro- topic has a status, which shows whether the topic has been passed, is
cedure detects changes in long-term individual learning preferences and in-progress, or forthcoming. For each in-progress topic, one educational
results in more relevant content suggestions. It should be mentioned resource is recommended (displayed). Besides accessing (and learning)
that the values of the long-term vector can be also viewed and directly the content, the learner has the following options with respect to the
edited by learners through their dashboard, in their profile settings. recommended learning content:

3.5.2. Recommendation engine • Change: If the learners are not satisfied with the content for some
To recommend learning content on a specific topic to a particular reason (e.g., it is not relevant, instruction does not fit the pref-
learner, first we retrieved all the resources (the ones that passed our erence, the format of the content is not preferred, low technical
filtering process) which focused on the topic. Afterward, we created the quality of the video/audio/text), they can replace the presented
same 15-dimensional vector (with the same features as the preference learning content, with another one addressing the same topic, at
vector) for each retrieved learning resource, as we did for the learners the same level. Thus, the recommendation engine records this
(see Section 3.5.1).8 Finally, we calculated the Dot Product [42] of the Change command as an instance of feedback with a minimal value.
learner’s long-term preference vector together with the created vectors At the same time, it updates the learners’ short-term preference
of each retrieved learning content. As a result, our system recommends vector as described in Section 3.5.1, and provides an alternative
the content with the highest Dot Product result. educational resource, on the basis of the updated vector.
• Done: When a learner completes a specific learning resource, they
3.6. Learning dashboard can indicate that with the Done button, and optionally rate the
learning content on a 5-point rating scale. The learner’s profile
In this section, we showcase our learning dashboard, called eDoer,
is automatically updated based on this rating, as described in
that we implemented to provide our individualized learning services
Section 3.5.1. Learners can also indicate whether they would
3.1.4 to learners.9 Fig. 1 illustrates how the different technical com-
like another learning content on the same topic, or whether they
ponents of our recommender prototype interact with one another (and
would like to try to progress to the next learning topic(s) related
with the learner) to create the learner’s personal learning experience
3.1.4. For the User Interface (UI) we incorporated responsive web to a particular skill target by (successfully) taking a progress
design and design guidelines [43,44]. We provided learners with an assessment (see 3.4.4).
interactive tutorial [45] at their first login, in order to familiarize them
with the different functionalities of our learning dashboard.
10
https://github.com/ali-faraji90/edoer/blob/main/Files/
RegistrationForm.png.
8 11
As an example, for a Short content, we set the Length-Short feature to 1, https://github.com/ali-faraji90/edoer/blob/main/Files/GoalSelection.
and the Length-Medium and Length-Long features to 0. png.
9 12
https://github.com/ali-faraji90/edoer/blob/main/Files/Demo.mp4?raw= https://github.com/ali-faraji90/edoer/blob/main/Files/Preferences.png.
13
true. https://github.com/ali-faraji90/edoer/blob/main/Files/Curriculum.png.

7
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

Fig. 1. Interaction between different parts of our prototype dashboard to provide the required services.

3.6.3. Search page 4. Validation

In order to address Service_1 in Section 3.1.4, we provided a straight-
forward and simple way for learners to search through all of the open
and available learning resources that are accessible on our platform. In an effort to demonstrate the effectiveness of our proposed online
Search results (a particular learning resource) can be added to the open educational resource recommender (eDoer), below we report on
curriculum page, and they are displayed as extracurricular learning a randomized experiment carried out with the explicit aim of having
resources. real users interact with our prototype. The experiment was conducted
to support the internal validity of our system, by answering the question
3.6.4. Monitoring learning objectives of whether engagement with our system results in improved knowledge
To address Service_4 (see 3.1.4), we implemented a Goal Page for acquisition. In this part, we showcase the methods and results of this
learners to gauge their learning progress towards their skills targets. validation step.
The page, therefore, provides detailed information on the number of
completed learning topics for each skill.14 It should be mentioned that
changing the target job, or removing a skill from the skill targets may
4.1. Objective
remove incomplete skill training curricula from the curriculum page.
However, learners can view both the new or updated target skills, and
also those skills that have been removed or are incomplete. Learners As mentioned earlier, to more formally evaluate eDoer with a par-
can also reactivate incomplete skills by means of a simple click. ticular focus on evidencing the internal validity of our inferences
Moreover, monitoring learning processes require attention to As-
pertaining to the effectiveness of eDoer in imparting knowledge in
sessment, as we discussed under Service_4 in Section 3.1.4. For this
a sample of students, we set out to conduct an experiment in the
reason, we deployed an Assessment Page to structure and keep track of
context of learning about statistics. Specifically, we formulated and
all skill assessments explained in Section 3.4.4. On this page, learners
can see and engage with comprehensive skill assessments for each of tested the following hypotheses, which were, by and large, premised
their target skills. Furthermore, to plot individual performance on skill on the fact that we specifically developed eDoer to address the most
assessments over time, a skill assessment history for each target skill is important requirements signaled by key stakeholders (see 3.1). Relying
provided as a graph. on an experimental design, enhanced our ability to rule out alternative
explanations for any observed effects.
3.6.5. History page
This page contains all learning resources that have previously been 1. Hypothesis 1. Using eDoer, as opposed to self-directed online
recommended to the learner. This gives learners the opportunity to search for open educational resources to learn about basic statis-
review any of these learning resources at will. Content on the history tics, has a positive effect on knowledge of basic statistics.
page is also categorized based on learners’ target skills and topics. 2. Hypothesis 2. Having eDoer provide personalized recommen-
Learners can also find information about their feedback regarding dations in terms of educational format (webpage, video, book,
learning content, including a timestamp of completion. slide), length (short, medium, long), level of detail (low detail,
medium detail, high detail), and content type (including exam-
3.6.6. Profile page
ple/theory or not), as opposed to having eDoer provide random
This page provides access (read and edit) to all the data we collected
content (from the quality controlled materials), has a positive
during the registration process (see Section 3.6.1) and beyond. This
effect on knowledge of basic statistics.
includes all demographic data, the target job, target skills, and learning
preferences.
Although these hypotheses are limited in their breadth and cov-
erage of the eDoer system, we feel they address the core functionali-
14
https://github.com/ali-faraji90/edoer/blob/main/Files/Goals.png. ties/requirements that we wanted to evidence at this stage.

8
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

4.2. Procedure directed to the curriculum page to start studying each

of the topics for the target skill within the defined time
For this experiment, we used the Prolific platform15 which is a period.
commercial service provider for connecting researchers with partici-
3. Step 3: posttest. After the learning process, all groups were
pants. In light of financial constraints associated with compensating
directed the posttest which included the same number of ques-
respondents for their time (we paid each respondent 15.76 British
tions, on the same topics, and with the same level of difficulty.18
pounds – approximately 21.74 US dollars – for their time and effort),
This set of questions was also differently timed in the same
we set out to collect high-quality learning data from a total of 150
manner that we did for the pretest.
participants. For this purpose, we decided to recruit a total number
of 175 users as we predicted that we might need to remove some of 4. Step 4: Feedback survey. Finally, all participants completed a
the participants’ data for different reasons (such as technical problems short survey to provide us with feedback. Group 1 received a
and/or missing data). survey.19 about the steps they took to learn the topics on their
We selected ‘‘Basic statistics for engineers’’ as the target skill for this own. Groups 2 & 3 received a survey20 about their experiences
study and ran our topic extraction method on it which resulted in the using eDoer Also, all groups were asked a question about their
following seven topics: 1. central tendency measures (i.e., mean, median, impression of the study in general.
mode), 2. variance and standard deviation, 3. covariance and correlation, Upon examining the data, we decided to remove 14 participants
4. conditional probability and independent variables, 5. normal distribution, from our study as they had 1 (or less than 1) correct answers from all
6. linear regression, and 7. hypothesis testing, 𝑝-value, and confidence 14 questions. We did this to exclude those respondents who were not
interval. The reason that we selected this particular skill was to target a participating seriously in our experiment. Also, we removed the data
fundamental (engineering-related) skill while at the same time ensuring of 5 participants data because of technical issues they had faced during
the availability of open educational resources for those people assigned the study. In the end, Group 1 consisted of 53 participants, Group 2 of
to the control group (who would not be engaging with eDoer). 50 participants, and Group 3 of 53 participants.
In order to take part in this study, the potential participants needed
to complete the following steps: 4.3. Measures
1. Step 1: pretest. In the first step, all users participated in a
pretest16 on ‘‘Basic statistics for engineers’’ that assessed prior We calculated both scores (i.e., pretest and posttest scores) for each
knowledge of the aforementioned seven topics. The test included individual participant as the number of correct answers divided by the
seven questions (one question per topic) which were selected total number of questions per test. Subsequently, we computed our first
through a discussion between three experts. The experts were measure progress score by subtracting, for each participant, the pretest
also asked to define the required time for each question in a way result from the posttest result.
that if a participant knew a topic, he or she would have enough Additionally, through Step 4 (Feedback survey), we collected par-
time to answer the question in the allotted time period. ticipants’ opinions on a 5-point scale (1: lowest to 5: highest) on the
After completing the pretest, participants were randomly as- following items and converted their ratings into a number between 0.0
signed to one of the following groups (to which they remained and 1.0 (i.e., 1 as 0.0, 2 as 0.25, 3 as 0.5, 4 as 0.75, 5 as 1.0):
blind): • Group 1:
• Group 1: Self-directed learning using online searches, but – Availability of educational content
without any support from eDoer – Quality of educational content
• Group 2: Learning through eDoer without personalized – Satisfaction with the Prolific experiment
recommendations
• Group 3: Learning through eDoer with personalized rec- • Groups 2 & 3:
ommendations
– Personalization of content
2. Step 2: Learning process. In this step, the participants were – Quality of educational content
granted 105 min (15 min per topic) and instructions (according – Satisfaction with the Prolific experiment
to their assigned group) to study the aforementioned topics in – Suggesting eDoer to other learners
order to be able to answer a new set of questions. The questions
Finally, to quantify learner’s overall satisfaction with eDoer’s rec-
were on the same topics as the pretest and within the same level
ommendations, we decided to collect the Evaluative Ratings (on a 5
of difficulty. The instructions were as follows:
point-scale) for the recommended educational materials.
• Group 1: In the learning process, the participants were
presented with the 7 extracted topics for a finer grain 4.4. Analytical procedures and results
searchability. They were free to engage with any type of
educational content they could find (e.g., through online On the pretest,Group 1, Group 2, and Group 3 achieved average
searches, reading books, and watching educational videos). scores of 0.22, 0.24, and 0.20, respectively. As expected, the pretest
• Group 2 & 3: These groups received simple instructions showed that most participants had no previous experience with Statis-
on 1. how to log in to eDoer using information from pre- tics before the experiment as their scores appear to reflect random
registered new test-users, 2. fill the preference form on responding. Also on the posttest, Group 1, Group 2, and Group 3
eDoer,17 and 3. adding the skill ‘‘Basic statistics for engi- achieved an average score of 0.34, 0.42, and 0.42, respectively. Based
neers’’ to their learning profile. Subsequently, they were on the pretest and posttest scores, we calculated our first measure as
progress score which showed how each group improved their knowledge

15
https://www.prolific.co.
16 18
https://uvafeb.eu.qualtrics.com/jfe/form/SV_5AudD6pyhqWb5vU. https://uvafeb.eu.qualtrics.com/jfe/form/SV_4Sl8QGDg5AtECSq.
17 19
Although Group 2 were not receiving personalized material, they also https://docs.google.com/forms/d/e/1FAIpQLSeEV5ekM6rAn_
filled out the preference form as they had not any information about which s0AscxTawgbVPm3eXjhwfF3Vjrqos_2HmnUg/viewform?usp=sf_link.
20
group they were assigned to. https://tib.eu/umfragen/index.php/887411?lang=en.

9
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

Table 5
Results of the eDoer evaluation experiment.
Measures Mean (out of 1) Standard deviation
Group 1 Group 2 Group 3 Group 1 Group 2 Group 3
Progress-score 0.12 0.18 0.22 0.20 0.19 0.17
Availability of educational content 0.56 – – 0.16 – –
Quality of educational content 0.64 0.75 0.82 0.27 0.22 0.20
eDoer recommendations’ rating – 0.79 0.87 – 0.17 0.16
Satisfaction from the experiment 0.74 0.90 0.90 0.23 0.14 0.14
Suggesting eDoer to other learners – 0.74 0.76 – 0.23 0.22

in the target skill on average. This measure was 0.12 for Group 1, 5. Discussion
0.18 for Group 2, and 0.22 for Group 3. As one can see Group 3,
which benefited from both eDoer and personalized recommendations, 5.1. Summary of the findings
showed the most improvement. Group 2 which benefited from eDoer
but received random (non-personalized) recommendations also showed Our system aims to support learners through labor market driven
some degree of improvement. Finally, and as expected, Group 1 which intelligent models that (1) match jobs with their required skills 3.2,
did not engage with eDoer had the lowest progress score. (2) decompose skills into meaningful components (topics) 3.3, and (3)
To formally test our hypotheses, a one-way ANCOVA21 was con- recommend high-quality open educational content to cover each topic
ducted. After controlling for the pretest scores, there was a statistically 3.4.2, as the key required features for learners based on the outcome
significant difference in posttest scores between the groups of learners, of our requirement analysis (See RQ1 and RQ2). Moreover, we showed
𝐹 (1, 152) = 11.202, 𝑝 < 0.001. Further investigation through pairwise that by using our 1. topic based, 2. metadata based, and 3. quality
comparison of estimated means showed that there was a statistically based prediction models, we can filter out the low quality educational
significant difference 𝑡(152) = 2.31, 𝑝 < .05 between the posttest materials in order to recommend quality resources to the learners (See
scores of the group receiving eDoers’ non-personalized recommenda- RQ3). Using the aforementioned components, we implemented our
tion (𝑀 = 2.91, 𝑆𝐷 = 1.18) and the group of self-directed learners (𝑀 = recommender prototype and made it accessible for learners through
2.38, 𝑆𝐷 = 1.16). Furthermore, our findings also show a significant a dashboard (See RQ4). We also evaluated our prototype through an
difference 𝑡(152) = 2.49, 𝑝 < .05 in test scores between self-directed experiment in the context of a fundamental engineering skill (i.e., Basic
learners and learners receiving eDoers’ personalized recommendations Statistics). This validation indicated that learners benefited from receiv-
(𝑀 = 2.98, 𝑆𝐷 = 1.27). However, there was no significant difference ing recommendations (see Hypothesis 1), and particularly so when such
between the posttest scores of the groups receiving non-personalized recommendations were personalized as evidenced by higher scores on
or personalized recommendations from eDoer 𝑡(152) = 0.137, 𝑝 = .892. the posttest of group 2 and 3 combined (eDoer groups), as compared
In support of hypothesis 1, our findings show that participants with the self-directed learning group 1.
who used eDoer without personalization attained significantly higher The hypothesized (see hypothesis 2) findings for the difference
scores on the posttest than participants who engaged in self-directed between learners who received personalized content as opposed to
learning (i.e., those who did not use eDoer). Unsurprisingly, and further those who received non-personalized content (i.e., randomly selected
supporting hypothesis 1, participants who used eDoer with personal- content) were less convincing, in that our most conservative test of this
ization also attained higher scores on the posttest than participants hypothesis, failed to reach statistical significance. Having said that, we
who engaged in self-directed learning. In contrast, no support was should remind ourselves that personalization is but a feature of our tool,
found for Hypothesis 2, in that there appeared to be no significant and that based on the findings for hypothesis 1, we may conclude that
difference in the posttest scores between those receiving personalized it made a difference to students’ learning, despite the effect pertaining
recommendations and those receiving non-personalized recommenda- to the difference between the personalized and non-personalized group
tions, again after controlling for scores on the pretest. To account not reaching statistical significance. When it comes to the lack of
for capitalization on chance, we reran the pairwise comparisons of support for hypothesis 2, one explanation is that both the personal-
estimated means applying a more conservative Bonferroni correction. ization and the non-personalization group received quality content,
The results of these analyses indicated a significant difference in the and that in some instances members of the non-personalization group
expected direction between self-directed learners and learners receiv- may in fact have received personalized content by chance (according
ing personalized recommendations 𝑡(152) = 2.49, 𝑝 < .05 but the to the limited number of educational resources that were offered for
difference between self-directed learners and learners receiving non- each topic). This would mean that those members contaminated what
personalized recommendations failed to reach statistical significance ought to have been an all non-personalization group with some degree
𝑡(152) = 2.31, 𝑝 = .066, even though it was in the expected direction. of personalization, therewith reducing the effect size. Note that this
It should be noted, however, that the Bonferroni correction has been explanation does not work for hypothesis 1 because we are certain that
criticized for being overly strict. none of the members in the control group can have made use of our tool
Table 5 shows the results of the other measures incorporated in (eDoer), hence preventing such undesirable diffusion of treatment.
our study for each group. In eyeballing these data, it is noteworthy
that ratings provided are most favorable for the personalized version of
5.2. Limitations and future work
eDoer, followed by the non-personalized version, and finally the self-
directed learning group. Moreover, the fact that 75% of the participants
The initial results of our validation are promising in that they
are willing to recommend eDoer to other learners, reflects their positive
seem to indicate that engagement with eDoer, particularly when it
attitudes towards the eDoer platform.
offers personalized recommendations pertaining to statistics, appears
to contribute to knowledge acquisition. Nevertheless, and as with all
21
We also used the Bayesian analysis [46] to test both of our hypotheses. The
research, clearly there are a number of limitations that need to be
reason that we also ran Bayesian hypothesis testing was to serve the interests acknowledged. First off, the sample size of our requirements gathering
of those who purport that Bayesian methods are superior [46]. However, the was quite limited, in that learners in different contexts, at different
results did not change the conclusions we derived based on the traditional levels, and of different ages, and from different cultures may have
t-test. different requirements that we have yet to learn about. Furthermore,

10
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

people with (learning) disabilities also have needs that are not ad- cannot provide the level of accuracy, which is needed for automatic
dressed by the current rendition of the system. A related challenge we decisions on content inclusion. Therefore, currently, we need to han-
faced in the requirements gathering process was how to reconcile free dle the errors that are produced by the models and minimize their
text input (in which we could qualitatively identify all the different impact manually. (2) Extracting properties from educational resources
requirements that learners felt needed to be addressed) with the rank- (currently done by manual labeling) is a time-consuming, and error-
ing of these same requirements (with which we could determine which prone activity. To tackle these problems, we plan to move towards
requirements were most important). Future work must be carried out to a crowdsourcing based quality monitoring and labeling strategy. This
identify and address these needs, particularly if eDoer is to contribute to will not only check the output of our intelligent models but will also
meeting the United Nations Sustainable Development Goal of providing improve our models (or help in building new models) based on the
inclusive and equitable quality education and promote lifelong learning participants’ (crowd) opinion.
opportunities for all, as was suggested in the introduction.
Despite the positive validation results there are also several issues 6. Conclusion
that are noteworthy with regard to our experiment. Our validation
comprised a limited sample of learners, studying but a single topic To remain employable, learners continuously need to master skills
for a very limited amount of time. It remains to be seen whether and topics that are relevant for their desired jobs in a dynamically
results will be equally promising when eDoer is deployed in different changing labor market. We initiated the work reported in this
contexts (for instance with unpaid learners, refugees, and/or those manuscript by conducting a requirement analysis to extract the learn-
seeking to qualify themselves for a new occupation), in other cultures, ers’ need for such a learning environment. Based on the results of our
with other learning content, and for a longer duration. To illustrate our analysis, we designed and implemented a system, called eDoer, that
point about duration, when we examined well-known courses on basic helps learners to set their learning goals and to receive a personal-
statistics from Standford University,22 The University of Amsterdam,23 ized learning path towards their goals. These learning paths contain
and Khan Academy,24 for instance, we determined that there students high-quality educational materials which have passed through our
spend an average of 10 h (600 min) to master the aforementioned automatic quality control models (i.e. topic based, metadata based, and
topics on basic statistics. Given that the current study established a quality based prediction models). We evaluated our prototype system
treatment effect for what constituted but a very limited ‘dosage’ of through an experiment in the context of a fundamental engineering
training, strengthens us in the belief that stronger effects can be booked skill (i.e. Basic Statistics). This validation showed tentative support for
with trainings of greater duration and depth. Clearly, however, future our first hypothesis, indicating that learners who used our system, per-
research is needed to further develop and evidence this tool, with formed better on a posttest than those learners engaging in self-directed
different samples, different topics, and training of greater durations. learning. The findings for the learners who received non-personalized
In addition to training duration, one may also wonder about the (i.e., randomly selected content) were less convincing, in that our most
longer term retention of that which was learned, in that our posttest conservative test of this hypothesis, which was about the difference
was administered quite soon after the training. Future research will between the personalized and non-personalized group, failed to reach
need to examine the extent to which that which was learnt is retained statistical significance.
over time. Here too, however, we feel that retention is only likely to
improve with trainings of greater duration. CRediT authorship contribution statement
Based on the feedback and the lessons we learned during the
prototype development process, we also conclude that more work Mohammadreza Tavakoli: Methodology, Software, Validation, Data
needs to be done on the personalization and scalability components Curation, Writing – Original Draft. Abdolali Faraji: Methodology,
of our prototype. Specifically, to personalize the learning experience, Software, Data Curation, Writing - Original Draft, Visualization. Jarno
we collected several initial personal features from learners (i.e. length, Vrolijk: Methodology, Validation, Writing - Original Draft. Moham-
level of detail, learning strategy, and content format 3.5.1). However, madreza Molavi: Validation. Stefan T. Mol: Conceptualization, Val-
this still needs to be extended to describe the learners’ context in a fine- idation, Supervision, Validation, Writing - Review & Editing. Gábor
grained manner. Therefore we see value in capturing more preference Kismihók: Conceptualization, Supervision, Writing - Review & Editing.
features in the future, such as language preferences, preferred authors,
location, or sensory information on learners’ cognitive and mental state Declaration of competing interest
(e.g., tiredness, well-being).
Moreover, currently, we use long-term and short-term vectors to The authors declare that they have no known competing finan-
plot learner preferences. At the moment, it puts more emphasis on cial interests or personal relationships that could have appeared to
their recent feedback about learning content they studied 3.5.1. In influence the work reported in this paper.
the user profile, however, learners can edit their long-term vector (the
basis for recommendations (see 3.5.2)) directly, which overwrites their Acknowledgments
preference score, computed by our model, based on actual learner
feedback and behavior (see 3.5.1). Therefore, we will need to fine-tune The authors gratefully acknowledge the financial support from the
this scoring algorithm by, for instance, providing an option for learners following projects that have helped in developing the eDoer plat-
to decide about the balance between their long-term and short-term form: ADSEE - Applied Data Science Educational Ecosystem, European
vectors. Commission - Erasmus Plus Programme, 2019-1-HR01-KA203-060984;
To arrive at a scalable open educational recommender system, we OSCAR - Online, open learning recommendations and mentoring to-
need to address two further issues: (1) Intelligent models, which we wards Sustainable research CAReers, European Commission - Erasmus
use in our educational content matching and content quality prediction Plus Programme, 2020-1-DE01-KA203-005713; BIPER - Business In-
steps, need to be very precise and accurate. Our algorithms sometimes formatics Programme Reengineering, European Commission - Erasmus
Plus Programme, 2020-1-HU01-KA226-HE-093987; ADAPT - Imple-
mentation of an Adaptive Continuing Education Support System in the
22
https://www.coursera.org/learn/stanford-statistics. Professional Field of Nursing German Federal Ministry of Education
23
https://www.coursera.org/learn/basic-statistics. and Research BMBF - INVITE 21INVI0501; WBsmart - AI-based digital
24
https://www.youtube.com/watch?v=uhxtUt_-GyM.&list= continuing education space for elderly care, German Federal Ministry
PL1328115D3D8A2566&ab_channel=KhanAcademy. of Education and Research BMBF - INVITE 21INVI2101.

11
M. Tavakoli et al. Advanced Engineering Informatics 52 (2022) 101508

References [24] A.R. Fabbri, I. Li, P. Trairatvorakul, Y. He, W.T. Ting, R. Tung, C. Westerfield,
D.R. Radev, Tutorialbank: A manually-collected corpus for prerequisite chains,
[1] United Nations, Goal 4 | department of economic and social affairs, 2020, survey extraction and resource recommendation, 2018, arXiv preprint arXiv:
https://sdgs.un.org/goals/goal4. 1805.04617.
[2] N. Sclater, Open educational resources: Motivations, logistics and sustainability, [25] A. García-Floriano, A. Ferreira-Santiago, C. Yáñez-Márquez, O. Camacho-Nieto,
in: Content Management For E-Learning, Springer, 2011, pp. 179–193. M. Aldape-Pérez, Y. Villuendas-Rey, Social web content enhancement in a
[3] V.B. Kobayashi, S.T. Mol, H.A. Berkers, G. Kismihók, D.N. Den Hartog, Text distance learning environment: intelligent metadata generation for resources, Int.
mining in organizational research, Organ. Res. Methods 21 (3) (2018) 733–765. Rev. Res. Open Distributed Learn. 18 (1) (2017) 161–176.
[4] M. Tavakoli, G. Kismihok, S.T. Mol, Labour market information driven, per- [26] S. Basu, Y. Yu, R. Zimmermann, Fuzzy clustering of lecture videos based on topic
sonalized, OER recommendation system for lifelong learners, in: International modeling, in: 2016 14th International Workshop On Content-Based Multimedia
Conference On Computer Supported Education (CSEDU), SciTePress, 2020. Indexing (CBMI), IEEE, 2016, pp. 1–6.
[5] F. Wang, Z. Jiang, X. Li, G. Li, Cognitive factors of the transfer of empirical [27] H. Shukla, M. Kakkar, Keyword extraction from educational video transcripts
engineering knowledge: A behavioral and fNIRS study, Adv. Eng. Inform. 47 using nlp techniques, in: 2016 6th International Conference-Cloud System And
(2021) 101207. Big Data Engineering (Confluence), IEEE, 2016, pp. 105–108.
[6] E. Colombo, F. Mercorio, M. Mezzanzanica, Applying machine learning tools [28] K. Horita, F. Kimura, A. Maeda, Automatic keyword extraction for wikification
on web vacancies for labour market and skill analysis, in: Terminator Or The of east Asian language documents, Int. J. Comput. Theory Eng. 8 (1) (2016) 32.
Jetsons? The Economics And Policy Implications Of Artificial Intelligence, 2018. [29] S. Aryal, A.S. Porawagama, M.G.S. Hasith, S.C. Thoradeniya, N. Kodagoda, K.
[7] V. Castello, E. Flores, M. Gabor, J. Guerrero, M. Guspini, J. Luna, L. Mahajan, K. Suriyawansa, Using pre-trained models as feature extractor to classify video styles
McGartland, I. Szabo, F. Ramos, Promoting dynamic skills matching: challenges used in MOOC videos, in: 2018 IEEE International Conference On Information
and evidences from the smart project, in: INTED2014 Proceedings, Citeseer, And Automation For Sustainability (ICIAfS), IEEE, 2018, pp. 1–5.
2014, pp. 2430–2438. [30] P. Király, M. Büchler, Measuring completeness as metadata quality metric in
[8] X. Li, Z. Jiang, Y. Guan, G. Li, F. Wang, Fostering the transfer of empirical Europeana, in: 2018 IEEE International Conference On Big Data (Big Data), IEEE,
engineering knowledge under technological paradigm shift: An experimental 2018, pp. 2711–2720.
study in conceptual design, Adv. Eng. Inform. 41 (2019) 100927. [31] A.R. Pelaez, P.P. Alarcon, Metadata quality assessment metrics into OCW repos-
[9] I. Wowczko, Skills and vacancy analysis with data mining techniques, in: itories, in: Proceedings Of The 2017 9th International Conference On Education
Informatics, vol. 2, (4) Multidisciplinary Digital Publishing Institute, 2015, pp. Technology And Computers, ACM, 2017, pp. 253–257.
31–49. [32] A. Romero-Pelaez, V. Segarra-Faggioni, P.P. Alarcon, Exploring the provenance
[10] V.B. Kobayashi, S. Mol, G. Kismihok, Labour market driven learning analytics, and accuracy as metadata quality metrics in assessment resources of OCW
J. Learn. Anal. 1 (3) (2014) 207–210. repositories, in: Proceedings Of The 10th International Conference On Education
[11] M.M. McGill, Defining the expectation gap: a comparison of industry needs and Technology And Computers, ACM, 2018, pp. 292–296.
existing game development curriculum, in: Proceedings Of The 4th International [33] A. Romero-Pelaez, V. Segarra-Faggioni, N. Piedra, E. Tovar, A proposal of
Conference On Foundations Of Digital Games, ACM, 2009, pp. 129–136. quality assessment of OER based on emergent technology, in: 2019 IEEE Global
[12] M. Tavakoli, A. Faraji, S.T. Mol, G. Kismihók, Oer recommendations to support Engineering Education Conference (EDUCON), IEEE, 2019, pp. 1114–1119.
career development, IEEE Frontiers In Education (FIE) (2020). [34] X.N. Lam, T. Vu, T.D. Le, A.D. Duong, Addressing cold-start problem in
[13] M.d.C. Saraiva, et al., Relationships among educational materials through the recommendation systems, in: Proceedings Of The 2nd International Conference
extraction of implicit topics: Relacionamentos entre materiais didáticos através On Ubiquitous Information Management And Communication, ACM, 2008, pp.
da extração de tópicos implícitos. [sn], 2019. 208–211.
[14] M. de Carvalho Saraiva, C.B. Medeiros, Finding out topics in educational mate- [35] M. Khobreh, F. Ansari, M. Fathi, R. Vas, S.T. Mol, H.A. Berkers, K. Varga, An
rials using their components, in: 2017 IEEE Frontiers In Education Conference ontology-based approach for the semantic representation of job knowledge, IEEE
(FIE), IEEE, 2017, pp. 1–7. Trans. Emerg. Top. Comput. 4 (3) (2015) 462–473.
[15] M. Molavi, M. Tavakoli, G. Kismihók, Extracting topics from open educational [36] D. Smith, A. Ali, Analyzing computer programming job trend using web data
resources, in: European Conference On Technology Enhanced Learning, Springer, mining, Issues Inf. Sci. Inf. Technol. 11 (1) (2014) 203–214.
2020, pp. 455–460. [37] P.G. Lovaglio, M. Cesarini, F. Mercorio, M. Mezzanzanica, Skills in demand for
[16] J. Djumalieva, C. Sleeman, An open and data-driven taxonomy of skills extracted ICT and statistical occupations: Evidence from web-based job vacancies, Stat.
from online job adverts, in: Developing Skills In A Changing World Of Work: Anal. Data Min. ASA Data Sci. J. 11 (2) (2018) 78–91.
Concepts, Measurement And Data Applied In Regional And Local Labour Market [38] E.M. Sibarani, S. Scerri, C. Morales, S. Auer, D. Collarana, Ontology-guided job
Monitoring Across Europe, 2018, p. 425. market demand analysis: a cross-sectional study for the data science field, in:
[17] J. Wang, J. Xiang, K. Uchino, Topic-specific recommendation for open education Proceedings Of The 13th International Conference On Semantic Systems, ACM,
resources, in: International Conference On Web-Based Learning, Springer, 2015, 2017, pp. 25–32.
pp. 71–81. [39] H. Jelodar, Y. Wang, C. Yuan, X. Feng, X. Jiang, Y. Li, L. Zhao, Latent Dirichlet
[18] M. Tavakoli, M. Elias, G. Kismihok, S. Auer, Quality prediction of open allocation (LDA) and topic modeling: models, applications, a survey, Multimedia
educational resources - a metadata-based approach, IEEE, 2020, Tools Appl. 78 (11) (2019) 15169–15211.
[19] M. Tavakoli, S. Hakimov, R. Ewerth, G. Kismihok, A recommender system for [40] M. Tavakoli, M. Elias, G. Kismihók, S. Auer, Metadata Analysis of Open
open educational videos based on skill requirements, IEEE, 2020, Educational Resources, in: LAK21: 11th International Learning Analytics And
[20] G. Sun, T. Cui, D. Xu, J. Shen, S. Chen, A heuristic approach for new-item Knowledge Conference, 2021, pp. 626–631.
cold start problem in recommendation of micro open education resources, in: [41] T.F. Hawk, A.J. Shah, Using learning style instruments to enhance student
International Conference On Intelligent Tutoring Systems, Springer, 2018, pp. learning, Decis. Sci. J. Innov. Educ. 5 (1) (2007) 1–19.
212–222. [42] Wikipedia, Dot product, 2020, https://en.wikipedia.org/wiki/Dot_product.
[21] J. Chicaiza, N. Piedra, J. Lopez-Vargas, E. Tovar-Caro, Recommendation of [43] Mozilla, Responsive design, 2020, https://developer.mozilla.org/en-US/docs/
open educational resources. An approach based on linked open data, in: Global Learn/CSS/CSS_layout/Responsive_Design#responsive_design.
Engineering Education Conference, IEEE, 2017, pp. 1316–1321. [44] Google, Material design, 2020, https://material.io/.
[22] S. Wan, Z. Niu, An e-learning recommendation approach based on the [45] Introjs, Introduce users to your product, 2020, https://introjs.com/.
self-organization of learning resource, Knowl.-Based Syst. 160 (2018) 71–87. [46] J.K. Kruschke, BayesIan estimation supersedes the t test., J. Exp. Psychol. General
[23] G. Sun, T. Cui, G. Beydoun, S. Chen, F. Dong, D. Xu, J. Shen, Towards 142 (2) (2013) 573.
massive data and sparse data in adaptive micro open educational resource
recommendation: a study on semantic knowledge base construction and cold
start problem, Sustainability 9 (6) (2017) 898.

12