See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/288823976

The e-health literacy framework: A conceptual framework for characterizing

e-health users and their interaction with e-health systems

Article in Knowledge Management & E-Learning An International Journal · December 2015

CITATIONS READS

178 12,095

7 authors, including:

Ole Norgaard Dorthe Furstrand

Copenhagen University Hospital – Steno Diabetes Center Copenhagen University of Copenhagen
33 PUBLICATIONS 1,126 CITATIONS 12 PUBLICATIONS 516 CITATIONS

SEE PROFILE SEE PROFILE

Louise Klokker Astrid Karnoe Knudsen

The Parker Institute University of Copenhagen
55 PUBLICATIONS 758 CITATIONS 17 PUBLICATIONS 725 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Ole Norgaard on 30 December 2015.

The user has requested enhancement of the downloaded file.

 Knowledge Management & E-Learning, Vol.7, No.4. Dec 2015

Knowledge Management & E-Learning

ISSN 2073-7904

The e-health literacy framework: A conceptual framework for

characterizing e-health users and their interaction with e-
health systems

Ole Norgaard
University of Copenhagen, Denmark
Dorthe Furstrand
University of Copenhagen, Denmark
Danish Cancer Society, Copenhagen, Denmark
Louise Klokker
Bispebjerg & Frederiksberg Hospital, Denmark
Astrid Karnoe
University of Copenhagen, Denmark
Roy Batterham
Deakin University, Melbourne, Australia
Lars Kayser
University of Copenhagen, Denmark
Richard H. Osborne
Deakin University, Melbourne, Australia

Recommended citation:
Norgaard, O., Furstrand, D., Klokker, L., Karnoe, A., Batterham, R., Kayser, L.,
& Osborne, R. H. (2015). The e-health literacy framework: A conceptual
framework for characterizing e-health users and their interaction with e-health
systems. Knowledge Management & E-Learning, 7(4), 522–540.
 Knowledge Management & E-Learning, 7(4), 522–540

The e-health literacy framework: A conceptual framework

for characterizing e-health users and their interaction with
e-health systems

Ole Norgaard
Department of Public Health
University of Copenhagen, Denmark
E-mail: [email protected]

Dorthe Furstrand
Department of Public Health
University of Copenhagen, Denmark
Danish Cancer Society, Copenhagen, Denmark
E-mail: [email protected]

Louise Klokker
The Parker Institute, Department of Rheumatology
Bispebjerg & Frederiksberg Hospital, Denmark
E-mail: [email protected]

Astrid Karnoe
Department of Public Health
University of Copenhagen, Denmark
E-mail: [email protected]

Roy Batterham
Faculty of Health
Deakin University, Melbourne, Australia
E-mail: [email protected]

Lars Kayser*
Department of Public Health
University of Copenhagen, Denmark
E-mail: [email protected]
Knowledge Management & E-Learning, 7(4), 522–540 523

Richard H. Osborne
Faculty of Health
Deakin University, Melbourne, Australia
E-mail: [email protected]

*Corresponding author

Abstract: In current e-health research and development there is a need for a

broader understanding of the capabilities and resources required for individuals
to use and benefit from e-health services, i.e. their e-health literacy. The aim of
this study was to develop a new conceptualisation of e-health literacy with
consideration of the experiences of a wide range of stakeholders and in
alignment with current technologies. Concept mapping was used to generate a
comprehensive and grounded model of e-health literacy. Concept mapping
workshop participants included patients, health professionals and medical
informatics experts. Eight workshops, carried out in Denmark and United
Kingdom, generated 450 statements, separated into 128 clusters. Through an
inductive structured analysis, seven domains were identified: 1. Ability to
process information, 2. Engagement in own health, 3. Ability to engage
actively with digital services, 4. Feeling safe and in control, 5. Motivation to
engage with digital services, 6. Having access to systems that work, and 7.
Digital services that suit individual needs. These empirically derived domains
form an e-health literacy framework (eHLF) and provide new insights into the
user’s ability to understand, access and use e-health technologies. The eHLF
offers a framework for evaluating an individual’s or a population’s capacity to
understand, use and benefit from technology to promote and maintain their
health. Such a framework also provides a potential checklist for the
development and improvement of e-health services.

Keywords: e-Health; Literacy; Framework; Interaction; Technology

Biographical notes: Ole Norgaard holds a Master of Science degree in public

health from University of Copenhagen. During his previous enrolment in a PhD
fellowship funded by the Danish Council for Strategic Research he investigated
the information behaviour of people living with chronic health conditions. The
focus of his research has primarily been on health literacy and how people can
use digital media to manage their health. He now works on e-mental health
projects at the Danish Veteran Centre.

Dorthe Furstrand is an MD and holds a Master of Science degree in health

informatics from University of Copenhagen. She is currently enrolled as a PhD
fellow at the Danish Cancer society, working with development of e-health
services from a user’s perspective.

Louise Klokker is a physiotherapist by background, now a PhD fellow at

University of Copenhagen, employed at the Parker Institute, Bispebjerg and
Frederiksberg Hospitals, Copenhagen. Louise takes a special interest in
scientific methodology and patient involvement in research.

Astrid Karnoe has a Master of Science degree in health informatics from

University of Copenhagen. She takes special interest in the design of digital
health services, e-health literacy and how to measure e-health literacy. She
currently works at University of Copenhagen in a research assistant position
funded by The Health Foundation in Denmark.
524 O. Norgaard et al. (2015)

Roy Batterham holds a B.App.Sci (physiotherapy) and a M.Ed (program

evaluation). He has worked for 20 years as a public health academic and
program evaluator in academic, private and government sectors. Roy has
developed many innovative methods and applications for Concept Mapping in
conducting more than 500 groups in multiple countries and languages. He
currently works on health literacy interventions in Thailand and South East
Asia.

Lars Kayser is a board certified specialist in internal medicine, associate

professor in health informatics and currently director of Health Informatics
study at University of Copenhagen. He has a particular interest in health
literacy, e-health literacy and innovative redesign of healthcare provision.

Richard H. Osborne is professor and the Chair of Public Health at Deakin

University, Australia. He holds a prestigious Australian National Health and
Medical Research Council (NHMRC) Senior Research Fellowship focusing on
Global Health Literacy Development and Equity. He initiated and led the
development of several questionnaires and tools used globally, including the
Health Literacy Questionnaire (HLQ) and the Ophelia (OPtimising HEalth
LItercy and Access) approach. The main focus of his research is the
development and implementation of processes to support healthcare
organisations and governments to strengthen health systems and reduce health
inequalities.

1. Introduction
Today, problems and challenges are often met with digital or technological solutions. The
health domain is not an exception to this phenomenon, with the e-health industry rapidly
expanding in many directions. The World Health Organization (WHO) defines e-health
as “the use of information and communication technology (ICT) for health” (World
Health Organization, 2015). e-Health is considered a way to improve quality, capacity,
efficiency and access to healthcare services and information (Hernandez, 2009), and thus
holds the potential to promote health (Camerini & Schulz, 2012) and improve health
equity (Dodson, Good, & Osborne, 2015; Neter & Brainin, 2012).
Engagement in one’s own healthcare, having access to health services and
handling health information in an appropriate way is a complex task for many people,
where the complexity is further increased with the addition of e-healthcare options. The
challenges incorporated in finding, understanding, using and appraising health
information and health services have been labelled as health literacy (Nutbeam, 1998).
Health literacy has been found to be strongly correlated with socioeconomic factors such
as educational level (Beauchamp et al., 2015; HLS-EU Consortium, 2012). Health
literacy, often measured using tests of health-related reading ability and numeracy, has
also been associated with a wide range of behavioural and health outcomes (Berkman,
Sheridan, Donahue, Halpern, & Crotty, 2011). Inspired by the concept of health literacy
and the potential for it to be a health mediator, Norman and Skinner have proposed the
term e-health literacy as a way to capture this emerging field (Norman & Skinner, 2006).
The many diverse aspects of e-health, from being a tool for providers to deliver
better care, to a way for individuals to be informed about their own health (Oh, Rizo,
Enkin, & Jadad, 2005), require that the knowledge, skills and resources needed for people
to use e-health tools are well understood. This understanding should inform the
Knowledge Management & E-Learning, 7(4), 522–540 525

development, selection, implementation and use of tools to ensure people receive

beneficial outcomes (Kayser, Kushniruk, Osborne, Norgaard, & Turner, 2015). To
generate such benefits, an e-health service, ideally, should enable users to strengthen the
skills, knowledge and resources they need to engage with the service at hand (Gilstad,
2014).
Norman and Skinner (2006) proposed the “Lily Model” for e-health literacy
including six types of literacies: traditional (reading ability and numeracy), information,
media, health, computer and science literacy depicted as petals of a lily. An expansion of
this model with a cognitive element, generated through the lens of Bloom’s taxonomy
was then proposed by Chan and Kaufmann (2011). More recently, Norman (2011) raised
the idea that the change in context and opportunities with Web 2.0 merits a rethinking of
the e-health literacy model. A further expansion to the model was suggested by Gilstad
(2014) which included elements such as communicative expertise, the bodily experience,
cultural, social and institutional context and propositional and procedural literacy. Further
work by Koopman, Petroski, Canfield, Stuppy, and Mehr (2014) included a larger set of
skills, motivation and concerns, together termed “readiness for health information
technology”, in the PRE-HIT instrument. This model was developed through focus
groups with patients and identified a series of themes hypothesised to have relevance in
predicting use of health information technology from a patient’s point of view. The
model also considered factors affecting willingness to engage in e-health, such as privacy
and security concerns, computer anxiety, desire for self-management, need for
information, convenience and perceived superiority of information from health
professionals.
To date, the research in this field has lacked systematic inclusion of users and e-
health professionals in the development of the e-health literacy concept. In this study, we
applied systematic inductive methods, seeking to identify the full range of elements
relevant to individuals attempting to use e-health technologies. The specific aim of the
research was to develop a model of e-health literacy which includes the views and
experiences of a wide range of stakeholders.

2. Methods
In order to develop a model of e-health literacy that is likely to resonate with IT users and
non-users, patients, healthcare providers, IT experts and managers, we conducted a series
of concept mapping workshops with this diverse range of stakeholders to capture a wide
range of perspectives. Concept mapping exists in many versions; we used a computer
assisted process originally developed by Trochim (Kane & Trochim, 2007; Trochim,
1989a; 1989b). This process has been used broadly for consulting with a wide range of
stakeholders to develop conceptual models for questionnaire development and is
described in detail elsewhere (Busija, Buchbinder, & Osborne, 2013; Klokker et al., 2015;
Osborne, Batterham, Elsworth, Hawkins, & Buchbinder, 2013; Osborne, Elsworth, &
Whitfield, 2007).
Eight workshops were completed between June and August 2012, four with
patients and four with professionals (see Table 1). All patients had a chronic health
condition and were recruited from a rheumatology clinic, a general practitioner and an
umbrella organisation for patients’ associations. The group of professionals included
health professionals, health informatics professionals, researchers in public health, and
computer scientists recruited through the authors’ networks. To maximise cultural and
system diversity two workshops were conducted in London, United Kingdom, with
 526 O. Norgaard et al. (2015)

professionals, including representatives from patient’s associations. The remaining

workshops were conducted in the Capital Region of Denmark.
Table 1
Workshop and participant characteristics: Overview of the eight workshops describing
the characteristics of the workshops, participants and the outcomes.
Work- Number of Age of Country of Type of Number of Number of
shop participants participants, the participants statements dendrogram-derived
mean (range) workshop clusters
1 9 n/a Denmark Professionals 46 16
2 12 n/a Denmark Professionals 62 17
3 7 n/a United Professionals 65 17
Kingdom
4 10 n/a United Professionals 61 18
Kingdom
5 3* 69 (68-71) Denmark Patients 47 15
6 6 57 (50-71) Denmark Patients 57 16
7 5 53 (42-64) Denmark Patients 59 15
8 6 59 (37-73) Denmark Patients 53 14
e-survey 23 n/a International Stakeholders 67 17
* Two researchers, in addition to the patients, did the sorting task, so that five people in total sorted the cards
during this workshop.
The first step in concept mapping uses a nominal group technique which is a
brainstorming process structured to maximise the breadth of ideas generated and to
achieve equality of input from participants. In this step, participants responded to a
seeding statement: “Thinking about citizens’ experiences in trying to look after their
health (or the health of their family), what does a person need to be able to do in order to
use digital health services?” To support participants to understand the context, the term
“health care” was defined as: “All places where you find treatment, rehabilitation, disease
prevention or health promotion, and at all levels, private, municipal, regional or state.”
Given the diversity of participants, a short sensitising discussion was held about what the
term “digital health services” meant. The discussion was seeded with “Think about
digital health services as all kinds of services in which digital technologies are used. It is
not restricted to traditional computer use”. This discussion ensured that all participants
had a broad understanding of the potential range of elements of this field.
A standardised method for conducting a concept mapping workshop was followed.
This included printing the statements generated by participants onto cards, which were
then sorted by participants in any way that made sense to them. These sorts were entered
into a computer program during the workshop and combined using multidimensional
scaling to produce a two-dimensional scatterplot (the “concept map”) in which statements
that were sorted together by many participants sit close together and statements that were
rarely or never sorted together are far apart. Hierarchical cluster analysis was then
undertaken to draw boundaries around groups of closely located items. Next, the map
was presented to the participants who were allowed to refine the map by re-assigning
items to different clusters and naming each cluster based on discussion among the group
of participants. As a final task, the participants were asked to rate the importance of each
statement on a five-point scale. The software used was The Concept System (version 1.0
by Trochim, 1987). This software applies non-metric multidimensional scaling using
Knowledge Management & E-Learning, 7(4), 522–540 527

Kruskal’s algorithm (Kruskal, 1964) and cluster analysis applying Ward’s method (Ward
Jr., 1963).
Dendrograms were used to identify commonalities across concept maps. These
explore the hierarchical structure of the data from broad concepts to increasingly refined
sub-concepts down to the individual statements provided by workshop participants.
Hierarchical cluster analysis was applied to produce a dendrogram for each workshop
showing all cluster solutions from two to 20 clusters. With these diagrams the division of
clusters were explored, looking at the specific item content each time a cluster is split into
two smaller clusters. As the aim of this study was a detailed and complete understanding
of the key elements of e-health literacy, clusters were increasingly split until the
maximum number of clusters that made conceptual sense to the researchers was reached.
These clusters, 128 in total, were labelled according to the content of their statements (see
Table 2). Finally, an analysis of the concept labels generated in the workshops and the
analysis of the dendrograms were consolidated into one overarching model of the key
hypothesized domains of e-health literacy.
As a validation, a concept mapping was subsequently applied through an e-survey
of health professionals across the networks of the research group, by e-mail and posted
on social media (selected LinkedIn groups). This type of e-survey, previously described
by Klokker et al. (2015), involved a two-step approach; (1) statement gathering and (2)
statement sorting. In the first step, participants were asked to provide up to 20 brief,
narrative statements responding to the same seeding statement as used in the face-to-face
workshops (outlined above). Further, participants were asked to provide demographic
data including their country of residence, profession, academic qualifications, and current
work role. Statements were collated, and duplicates and very similar statements removed.
In the second step, the statement pool was sent back to the professionals for sorting and
rating, and then analysed as described above.
All data obtained during workshops was anonymised and neither biological
samples nor medical equipment was used. Consequently, specific ethical approval was
not acquired under the regulations of the Danish National Data Protection Agency or the
Danish National Committee on Health Research Ethics in force during the data collection
period.

3. Results
The multidimensional scaling of the data collected during each workshop and the
resulting two-dimensional map with collectively labelled clusters informed the
subsequent analysis. An example of the output, a concept map from the first workshop
with professionals, is shown in Fig. 1.
From the experience of the eight workshops and the content analysis resulting in
128 cluster labels, seven main themes were hypothesised initially through an inductive
analysis by the first author (ON) followed by extensive discussions with RHO, RB, LOK
and LK. The cogency of the content and labels for the themes was then further discussed
and refined (ON, RHO, LK, DF, AK). The conceptual independence of each domain was
carefully considered such that each could be suitable for later development into a scale
within a multidimensional e-health literacy questionnaire.
The labels of the smallest meaningful clusters, resulting from the dendrogram
analyses, and the domains they were attributed to, are reported in Table 2, to illustrate the
 528 O. Norgaard et al. (2015)

ideas promoted by patients and professionals respectively. The content and distinctions of
each domain are derived from the statements belonging to these clusters.

Fig. 1. Example of a concept map from the first workshop held with professionals. Each
number represents a statement and each shape represents a cluster as calculated by the
software. The label of each cluster is proposed by workshop participants through
consensus.
Table 2
e-Health literacy domains and associated smallest meaningful cluster labels: Cluster
labels with similar content were merged to create greater overview of data, thus reducing
the number of cluster labels from 128 to 88.
1. Ability to process information
Patients: Professionals:  Ability to share information with
others
 Have the mental resources to cope  Cognitive abilities
with vast information  Basic/computer literacy
 Functional literacy
 Ability to read, write and learn  Understand the language used

2. Engagement in own health

Patients: Professionals:  Wanting to take responsibility of
one's own health
 Feeling confident that you can  Taking responsibility of own life
manage your own health  Wish to prioritize health
 Being able to navigate between
 Coping with your condition services
 Understanding of own condition
 Knowledge Management & E-Learning, 7(4), 522–540 529

3. Ability to actively engage with digital services

Patients:  Ability to progress in acquiring  Know how to use the systems in
skills general
 Readiness to start using the systems
 Sense of control over own health  Know how to use systems related to
 Critical thinking and the systems health
 Logical thinking
 Understand and appraise
 Be introduced to IT Professionals:
information

 Ability to learn how to use IT  Basic skills in using IT  Ability to utilize information and
system
 Know one's limitations and be open  Critical assessment skills
 Know how to critically interact
 IT may lower one's attention  Health literacy
 Have confidence in oneself
 Feel confident in systems and data  Feeling comfortable with what
information is used  Know other people's (hidden)
 Ability to use existing systems agenda
 Basic computer skills  Know when and how to get help
 Understand confidentiality
 Feel confident in using IT  Information exchange and feedback
from health care professionals and  Ability to access the systems
 Know how to access the systems systems  Accept non-human interface during
 Be able to make sense of data and interaction with health care
 Know how to read and write using
IT understand what it is used for  Confidence in using IT
 Ability to appraise information  Being able to use the IT systems  Incorporate use of IT into one's life
 Ability to learn the context-specific  Knowledge about your different  Deeper understanding of how
stills (IT) options systems work
 Cognitive abilities and training  Being critical when using the
information one gets from the system

4. Feel safe and in control

Patients:  Secure and stable systems  Know that data is secure
 Trust that your information is  Easy access to correct information  Data ownership
secure
 Access to help and support so you  Feeling of ownership of data and
 Security can act safely when using IT systems
 Feel that data are secure  Trustworthy information
 Ownership of personal data Professionals:  Trust that the systems are safe
 Data security  Feeling safe/secure
 Know where your data is  Feel that data is secure
 530 O. Norgaard et al. (2015)

5. Motivated to engage with digital services

Patients:  Feeling of distance (mentally)  Feel that use of IT is beneficial
created by IT
 Empowerment of patients  Benefits are made clear
 Having the energy to use new
 Have interest and cognitive ability  Need to be "pulled"
technology
 Ability to adjust own behaviour  Use of IT makes good sense
 Feeling unsure about new things Professionals:  Motivated to use IT
 Forced to use IT  Motivation  Ability to self-manage via IT
 Be motivated to learn  Having the courage and be curious

6. Access to digital services that work

Patients:  Easy access to systems and support  Trust that using IT is as good as the
personal (old) way
 Accessibility (to equipment, to  Access to systems that work
systems) efficiently (and work together)
 Access to relevant information  Correct use of registered Professionals:
information  Easy access to receiving and adding
 Ability to interact with the systems
 IT systems should work to be relevant information
 Access to help beneficial  Physical access (x2)
 Systems should be able to work
 Access to equipment (computers  Access to systems
together
etc.)
 Using IT may be high on resource  Easy access to information
 Having access to well-functioning
consumption services
 Physical access to IT
7. Digital services that suit individual needs
Patients:  Personal contact with health  Physical abilities (x2)
professionals
 Adaptable systems  Systems that can be adjusted to
individual needs
 Accessible systems
Professionals:  Access to training in system use
 Access to help (x2)
 Access to service that suits your  Access to information that suit
 Access to support needs individual needs
 Communication that fits the  Relatives  Access to the help and support that
individual
 Having power over the IT systems one needs
 Experience that IT helps you in (and the opposite way)  Empowerment of carers
managing your health
 Digital services that suit individual
 Access to help and support that suit needs
one's needs
Knowledge Management & E-Learning, 7(4), 522–540 531

1) Ability to process information

The processing of information has several steps, from recognizing the need for and
finding information, through reading and understanding, to the process of appraising and
applying information to one’s own situation. Each of these steps was present in
statements assigned to this domain. A core element of this domain is the “Ability to use
the written word” (professional) as well as basic numeracy, along with the notion of
logical thinking. Some participants suggested deeper cognitive skills, including having
“Cognitive ability to understand basic information” (professional), and the ability to
understand medical language: “Able to read medical information” (professional). Other
participants suggested issues related to accessing, clarifying and evaluating information:
“Knowledge of how to access” (professional); “Ability to discuss and share information
with others” (professional); and “You have to be smart, so you can filter the
information …” (patient). The domain also includes statements with focus on memory,
attention and having the strength to get involved in the information. This domain was
well represented in all workshops and was discussed as a fundamental skill, necessary for
all.
2) Engagement in own health
This domain included the basic knowledge about one’s own health conditions, how to
approach the healthcare system, as well as the approach to health in terms of will and
responsibility. All of these imply that the person has an interest in learning about and
managing their health. A basic level of engagement is seen in statements such as ”Basic
knowledge about your condition (symptoms or needs)” (professional), statements that
indicate some familiarity with health systems: “Be able to navigate in an authorised
organised healthcare system” (professional); and statements indicating an ongoing focus
on health: ”You should be able to take responsibility for your own disease” (patient); and
“Health has to be a priority in their lives compared with other issues” (professional).
There was also the ethical consideration, that there “Needs to be respect for people’s
personal choice to engage or not” (professional).
3) Ability to actively engage with digital services
This was a broad domain containing 153 statements. It contains not only the basic
knowledge and skills connected to the use of digital services, but also an important
element on how to process information and data in the framing of the digital media, thus
including the abilities needed to use these technologies in a healthcare setting. According
to statements, “You need competencies to use and understand information technology”
(professional), and experience from use of information technology in other contexts was
seen as a strength. Basic skills included “Understanding how to use search engines”
(professional), “Being able to navigate the Internet” (professional), and logic skills
appear across workshops. Higher skills were also noted: “Develop confidence to use the
technology” (professional) and have the “Ability to absorb new concepts” (professional).
The idea that “One should know the pitfalls on the Internet” (patient) is in line with the
reappearing notion that “You should have a healthy scepticism” (patient). Other skills
surrounding the use of digital services are responsibility: “Patience (to deal with
problems)” (professional) and to “Accept the technology into their home/lifestyle”
(professional).
4) Feel safe and in control
Consistently through all workshops, with both patients and professionals, the issue of
safety was mentioned. The statements ranged from feeling safe and trusting the systems,
532 O. Norgaard et al. (2015)

to the sense of security and control over data. “Feeling that your information is safe and
secure” (professional) appeared to be imperative to participants. Many statements
focused on the notion of trust: “Trust in the source of information” (professional) and
“You have to trust that the information you give is not misused” (patient). This reflects
the sensitive nature of health data and demonstrated that safety and security was a high
priority to participants. Further, it leads to the need of being in control and promotion of
user rights: “You should own your own data” (professional) and “You should know what
your data are used for” (patient), with transparency being a key value. The need for
personal contact was also raised for participants to feel safe: “Human contact is still
required” (patient). Commercial interests were raised but not frequently emphasised:
“Need to feel that the technology is not driven by commerce (being tracked – big brother
is watching you)” (professional).
5) Motivated to engage with digital services
Statements concerning the incentives for using e-health formed this domain, with a focus
on benefits aligned with use, but also including attitudes such as curiosity, courage,
enjoyment and feeling of closeness and comfort as motivational factors for use. The label
arose from statements such as “Citizen is motivated to use IT” (professional). The value
of the system to the user, the “Purpose or need to use the technology” (professional) was
a core feature, as was “Some people might need an incentive (a sense that they will get
something from it, e.g. might save money, link with others)” (professional). Some
statements included specific motivations such as “You should have courage”
(professional), “You should not be afraid to click around on the [web] pages” (patient),
“You should be curious” (professional) and “You should believe in yourself” (patient).
In this domain statements also touched on other aspects of the attitude towards
digital health, such as “Your age is of significance” (patient), that “Social isolation can be
a reason to use it” (professional) or a situation where it can be “Imposed upon you (you
will have to accept IT in the health system)” (patient), so you feel forced to use it,
resulting in “You can feel alienated” (patient).
6) Access to digital services that work
In this domain, the statements related to access to hardware and software. Statements
included “Access to devices (electronic and medical)” (professional), “You should have
access to solutions” (professional) and that “The system should be easily accessible (user
friendly)” (patient). A recurring point was that “You should be able to afford to buy and
use the technology” (patient). The need for flexibility was also evident, i.e. the need to be
“Able to have 24/7 access” (professional) and that “The systems should communicate
with each other (integration)” (patient). A broad understanding of access emerged,
including “Access to help/manuals” (patient). Not only the user needs access, there was
also an aspiration for “Health information kept in one place where you and relevant
health care professionals have access” (professional). The statement “There are fewer
errors when the patient helps to register drugs and test results” (patient) suggested why
this was important to participants and supports that “The solutions should work”
(professional).
7) Digital services that suit individual needs
This domain focused on the system matching the needs of the individual user. It includes
the user interface: “The user interface must match the user (the user must help to
develop)” (patient) and “There should be understandable feedback from the health
system” (patient). Also it should be “Available in my language” (professional) and “You
should be able to understand the messages you get (not medical language)” (patient).
Knowledge Management & E-Learning, 7(4), 522–540 533

Essentially, “The system should be able to be fitted to the citizen (an adaptive system)”
(professional) and “The system should support the citizen’s development (from novice to
experienced user)” (professional). Data should be adapted to fit the receiver, so that it
provides “Access to having results (e.g. a blood test) interpreted” (patient). An important
aspect of this domain was the systems adapting to disabilities, such as “Options for
blind/visually impaired” (patient) and the option of including personal networks when
needed, in the form of “Empowering relatives when citizens are not able” (professional).
Finally, the system should be receptive to the user’s rights, such as “Having the right not
to know” (professional) and “Acknowledgement that there will be people who will not
use the technology” (professional).
Confirmatory e-consultation process
The e-consultation resulted in 67 unique statements, which were broadly consistent with
the seven domains, and all domains were represented in the data. The health aspect in the
second domain, “Engagement in own health”, was represented by statements such as
“Understand health concepts (bodily functions etc.)”. The fourth domain: “Feel safe and
in control”, was illustrated by the statement “Trust the source of information”. The fifth
domain on being motivated to engage with digital services was included in statements
such as “Have an incentive for using the digital health service.” Thus the e-consultation
confirms the results as presented above.
The e-health literacy framework (eHLF)
The seven domains were clearly related to each other and connect the individual and the
individual’s experience with the system acting as an e-health literacy framework. Fig. 2
represents the seven domains on a pair of axes: from domains largely dependent on the
individual to domains largely dependent on the system, on the horizontal axis; and from
domains that relate to externalized, observable actions to domains representing more
internalised concepts and feelings on the vertical axis. The first two domains concerning
ability to process information (domain 1) and engagement in own health (domain 2) were
largely dependent on the competences of the individual and thus placed on the
individual’s side in the model. Access to working systems (domain 6) and a system’s
ability to suit individual needs (domain 7) depend mainly on the characteristics of the e-
health systems, so these two domains were placed on the system side of the model. The
user skills and knowledge have little influence on whether there is hardware accessible or
an Internet connection when needed, or whether the system adapts to individual needs,
such as visual impairment, if the system is not providing this option. However, individual
capabilities and resources do influence perception of access to a system and to what
extent the system adapts to individual needs. Therefore, knowing about the user’s
perception of the technology contributes to an inclusive understanding of e-health literacy.
The interaction between the individual and the system is where unique aspects of
the concept of e-health literacy start to unfold. How a person might engage with
information in the context of a system (domain 3) is dominated by more than just
technical skills. Experiencing safety and control (domain 4), benefit and comfort, and
having the right attitude in approaching technology (domain 5) become just as relevant as
knowing the inner workings of the systems and having the skills to navigate it.
Moreover, the model includes the relative level of internalization of each ability.
Within each circle the more external skills, such as reading or having access to a working
system, were placed in the upper regions, whereas the more internalized concepts such as
motivation, engagement and individualisation were placed in the lower regions.
534 O. Norgaard et al. (2015)

Fig. 2. The e-health literacy framework (eHLF). The figure shows the interaction
between individual and system as illustrated by the domains.

4. Discussion
This study presents the development of a model of e-health literacy based on the
experiences of a wide range of stakeholders, by applying a systematic, inductive
approach. This has resulted in a seven-domain framework, the e-health literacy
framework (eHLF), which provides a new way to understand the interaction and relation
between individuals and the system.
The strength of the eHLF is that it includes not only the known domains of health,
information and basic digital skills, but also includes new elements with relevance to the
dynamics that occur when the system meets the individual, as exemplified by the
domains 3. Ability to actively engage with digital services, 4. Feel safe and in control,
and 5. Motivated to engage with digital services, placed in the interaction between these
two (see Fig. 2). Together with the domains 6. Access to digital services that work and 7.
Digital services that suit individual needs, bound to the system, this illustrates how the
eHLF supports an understanding of e-health literacy that is not only about the
individual’s abilities and resources but also strongly context dependent and related to the
complexity of systems.
 Knowledge Management & E-Learning, 7(4), 522–540 535

The eHLF is a new contribution to the debate on how electronic, digital or

technology health literacy can be understood. Since Norman and Skinner proposed their
model almost a decade ago, only one other study has contributed to the evolution of the
understanding of e-health literacy with a new model (Koopman et al., 2014; Norman &
Skinner, 2006). Table 3 compares the Norman and Skinner and Koopman models to the
seven domains of the eHLF, according to the similarities and differences of the elements
in the models. Some elements have been mapped to more than one of the domains in
eHLF, while others exceed the scope of the eHLF.
Table 3
Similarities and differences between eHLF and two other e-health literacy models by
Norman and Skinner and Koopman respectively
eHLF (domains) Norman & Skinner (literacies) Koopman et al. (themes)
Domain 1 Information literacy Poor computer and search abilities
Ability to process Traditional literacy Need for information
information Media literacy
Domain 2 Health literacy Desire to be a more active participant
Engagement in own Science literacy in own care
health Anxiety about what information might
be found on the Internet
Domain 3 Computer literacy Poor computer and search abilities
Ability to actively Science literacy Convenience
engage with digital Media literacy
services
Domain 4 Privacy and security concerns
Feel safe and in control
Domain 5 Preference for the health care team as a
Motivated to engage source of information
with digital services Anxiety about what information might
be found on the Internet
Desire to be a more active participant
in own care
Convenience
Domain 6 Asynchrony
Access to digital
services that work
Domain 7 Looking for information for or about
Digital services that others
suit individual needs
Not covered by the Preference for the health care team as a
seven eHLF domains source of information
Looking for information for or about
others
536 O. Norgaard et al. (2015)

All literacies included in Norman and Skinner’s Lily Model are contained in the
first three domains of eHLF that describe basic skills to process information, understand
health and use technology, although both eHLF domains concerning engagement are
broader. The content of the remaining four domains of eHLF are not included in Norman
and Skinner’s model.
The nine themes suggested by Koopman et al. are all represented in the eHLF
domains. Compared to the domains of eHLF, the themes from Koopman et al. tend to be
more specific. E.g. the theme “Asynchrony”, attentive to the digital influence on
increasing asynchrony in the access to healthcare professionals, but not regarding the
access to hardware, software, training and economic means, that are also included in the
eHLF domain.
Two of the Koopman themes mapped into domains 5 and 7 were only partially
covered in eHLF while most other themes are partially but not fully contained by the
eHLF. For example, the theme “Looking for information for and about others” is about
an individual’s need for information (domain 7), but it also contains a specific need for
looking up information on conditions not affecting one self.
Interestingly, the elements incompletely covered by the eHLF are already well
defined in broader understandings of health literacy. Our model is limited to the users
interacting with technology and digital services in a health context while the current
understanding of the concept of health literacy has multidimensional characteristics
(HLS-EU Consortium, 2012; Osborne et al., 2013), and includes aspects of social support
networks and engagement with health professionals.
As the only e-health literacy model for almost a decade, Norman and Skinner’s
Lily Model became the foundation for another two expansions of the original model
(Chan & Kaufman, 2011; Gilstad, 2014): Chan and Kaufman applied Bloom’s taxonomy
to each of the six literacies adding a cognitive dimension to the existing model (Chan &
Kaufman, 2011). The eHLF does not contain this taxonomic construct, most likely due to
its origin from a concept mapping-based development process. However, this could be
incorporated into the development of instruments to measure e-health literacy based on
eHLF.
In 2014, Gilstad suggested an extension of Norman and Skinner’s Lily Model
(Gilstad, 2014). Through an interdisciplinary review of technology studies, human and
social sciences and health studies she applied elements of communicative expertise, the
bodily experience, cultural, social and institutional context and propositional and
procedural literacy. These additions are not covered by eHLF but the context, relationship
and personal values may be covered in the understanding of health literacy or should be
included in a third dimension placed outside the interaction between the individual and
the system.
The concept mapping process that we undertook included a broader range of
stakeholders than was reported for the other models. The other models focus on
individual skills and none include the system domains that are represented in our model.
The two domains attributed to the system: 6. Access to digital services that work and 7.
Digital services that suit individual needs, include important features of what is needed
for the user to use and benefit from a digital service. The grounded approach with
inclusion of a wide range of stakeholders across settings is a strength of the eHLF
development that has generated new and highly relevant aspects of e-health literacy that
have the potential to advance this field.
Knowledge Management & E-Learning, 7(4), 522–540 537

The eHLF is a comprehensive framework, as it can be applied on both the micro,

meso and macro level. At the micro level, key factors for successful implementation of e-
health services include involving the user and understanding the users’ competences. A
lack of focus on these two may result in less effective systems or risk of project failure
(Cresswell & Sheikh, 2009; Greenhalgh, Hinder, Stramer, Bratan, & Russell, 2010; Lluch,
2011). There is a need for a framework guiding developers to better understand and meet
the needs of consumers.
At the meso level, the seven domains contain a broad range of features relevant to
planning, design and development of e-health services (Kayser et al., 2015). Inclusion of
users in the process of creating e-health tools is becoming an accepted practice (Kayser et
al., 2015; Roehrer, Bjørnes, Cummings, & Nøhr, 2014). However, there is a need for a
framework to ensure that all elements are systematically considered.
The concept mapping process has successfully been used as a technique for
validity-driven questionnaire development (Osborne et al., 2013; Osborne, Elsworth, &
Whitfield, 2007). The authors are currently using the model presented here to guide the
development of new instruments for measuring e-health literacy. The possibilities for
such a tool are many, including assessment of requirements of potential users and
evaluation. It could also be used to analyse the characteristics and strengths and
weaknesses of different digital services, and to identify socioeconomic factors associated
with limited e-health literacy.
The eHLF does not, however, need to be operationalized as a questionannire to
prove useful in research. Working with qualitative methodologies, for example a project
looking into how a specific digital health service was received, can use the framework as
interview or observation guide, to ensure relevant elements are included.
On a macro level, decision makers often use specific, quantifiable markers to
guide the commissioning of new digital services or strategies. This framework has the
potential to provide policy makers with a structured approach to include the user
perspective in their decision making.
In the near future, cooperating with digital services will become a central skill for
all health professionals. In their education and training, a deep understanding of the
factors involved in e-health literacy will become increasingly important and the eHLF has
the potential to become a framework for studying this field.
In this study, e-health literacy was approached with as broad a perspective as
possible, including allowing completely new ideas to be able to emerge through the
concept mapping process. Breadth was encouraged through the diversity of the
participant panels, the open brainstorming environment at the workshops and the
inductive approach to the analysis.
Most of the 450 statements gathered throughout the workshops, fitted well within
the seven domains. Some statements were central to the theme of the domain whereas
others were more peripheral. Other aspects are generic such that they relate to multiple
domains. For example, the need for training, which is a part of not only 3. Ability to
actively engage with digital services, but also a part of 4. Feel safe and in control, as well
as 7. Digital services that suit individual needs.
Diversity in cultural perspectives and health systems was facilitated by
conducting consultations in two countries – Denmark and United Kingdom. Both
countries are within western European culture. While we expect the eHLF to be robust in
538 O. Norgaard et al. (2015)

these and similar setting, the framework may require refinements in other contexts, such
as cultures from outside the European continent or in developing countries.
The background of the participants was wide ranging. The recruitment was
through convenience sampling, mainly via our own networks (professionals and e-survey
participants) and patients were recruited from clinics but also through patient associations.
Importantly, the structure of the concept mapping workshops ensured that all participants
felt safe to contribute their own view, with careful attention to not let the facilitators’
perspectives influence participants. A wider demographic spread among the participants
may have generated a wider set of responses. While the concept mapping provides some
reassurance that the eHLF domains are comprehensive, further work in this area is
warranted, particularly with individuals from non-European backgrounds.
The process of constructing a concept like e-health literacy will always involve
multiple analytic and decision-making stages. The systematic approach provided by the
concept mapping method and software contributed structure to the data, that supported
the inductive work of identifying themes among the statements. Being qualitative work,
the prior understanding of the researchers will influence the analysis. Thus the authors’
background in public health as well as medicine, education, and medical informatics, will
have an impact on the results. The validity and rigor of the qualitative process was
strengthened by repeated immersion in the content of the statements in multiple sessions
with different groups of authors as well as the confirmatory process of the e-survey. In
this work, the method facilitated the emergence of domains not previously recognized
from the literature that were clearly relevant to participants in the workshops.

5. Conclusion
This paper presents a new e-health literacy framework (eHLF) which provides a
comprehensive scaffold for investigation of an individual’s ability to use and benefit from
e-health technologies at the micro, meso and macro level. The seven domains provide
novel insight into e-health literacy from the perspective of the system and the individual
as well as the interaction between the two. The eHLF provides a starting point to assist
researchers, practitioners, funders and policymakers to explore e-health literacy from a
development, research, policy or user point of view in order to obtain a better match
between the demands of e-health systems and the knowledge, skills, resources and
motivation of users.

Acknowledgements
Parts of the project took place when LK and ON were funded by the Danish National
Strategic Research Council. RHO was funded in part by a National Health and Medical
Research Council of Australia Senior Research Fellowship. DF is a PhD fellow supported
by the Tryg Foundation. Author AK was funded by The Health Foundation in Denmark
for a six month project regarding e-health literacy. Mr. Jacob Ø stberg Hansen provided
assistance in collating statements gathered through the e-survey. Thank you to all who
contributed to this work by participating in the workshops or the e-consultation. Thanks
to Ms Lena Sundby Jensen for assisting in editing the paper.
Knowledge Management & E-Learning, 7(4), 522–540 539

References
Beauchamp, A., Buchbinder, R., Dodson, S., Batterham, R. W., Elsworth, G. R., McPhee,
C., Sparkes, L., Hawkins, M., & Osborne, R. H. (2015). Distribution of health literacy
strengths and weaknesses across socio-demographic groups: a cross-sectional survey
using the Health Literacy Questionnaire (HLQ). BMC Public Health, 15(1): 678.
Berkman, N. D., Sheridan, S. L., Donahue, K. E., Halpern, D. J., & Crotty, K. (2011).
Low health literacy and health outcomes : An updated systematic review. Annals of
Internal Medicine, 155(2), 97–107.
Busija, L., Buchbinder, R., & Osborne, R. H. (2013). A grounded patient-centered
approach generated the personal and societal burden of osteoarthritis model. Journal
of Clinical Epidemiology, 66(9), 994–1005.
Camerini, L., & Schulz, P. J. (2012). Effects of functional interactivity on patients’
knowledge, empowerment, and health outcomes: An experimental model-driven
evaluation of a web-based intervention. Journal of Medical Internet Research, 14(4):
e105.
Chan, C. V., & Kaufman, D. R. (2011). A framework for characterizing eHealth literacy
demands and barriers. Journal of Medical Internet Research, 13(4): e94.
Cresswell, K., & Sheikh, A. (2009). The NHS care record service (NHS CRS):
Recommendations from the literature on successful implementation and adoption.
Informatics in Primary Care, 17(3), 153–160.
Dodson, S., Good, S., & Osborne, R. (2015). Health literacy toolkit for low- and middle-
income countries: A series of information sheets to empower communities and
strengthen health systems. New Delhi: World Health Organization, Regional Office
for South-East Asia. Retrieved from http://apps.searo.who.int/PDS_DOCS/B5148.pdf
Gilstad, H. (2014). Toward a comprehensive model of eHealth literacy. In Proceedings of
the 2nd European Workshop on Practical Aspects of Health Informatics. Retrieved
from. http://ceur-ws.org/Vol-1251/paper7.pdf.
Greenhalgh, T., Hinder, S., Stramer, K., Bratan, T., & Russell, J. (2010). Adoption, non-
adoption, and abandonment of a personal electronic health record: Case study of
HealthSpace. BMJ, 341: c5814.
Hernandez, L. M. (2009). Health literacy, eHealth, and communication: Putting the
consumer first: Workshop summary. Washington, DC: Institute of Medicine of the
National Academies, National Academies Press.
HLS-EU Consortium. (2012). Comparative report on health literacy in eight EU member
States - The European health literacy project 2009-2012. Vienna: Ludwig Boltzmann
Institute for Health Promotion Research.
Kane, M., & Trochim, W. M. K. (2007). Concept mapping for planning and evaluation.
SAGE Publications, Inc.
Kayser, L., Kushniruk, A., Osborne, R. H., Norgaard, O., & Turner, P. (2015). Enhancing
the effectiveness of consumer-focused health information technology systems through
eHealth literacy: A framework for understanding users’ needs. JMIR Human Factors,
2(1): e9.
Klokker, L., Osborne, R., Wæ hrens, E. E., Norgaard, O., Bandak, E., Bliddal, H., &
Henriksen, M. (2015). The concept of physical limitations in knee osteoarthritis: As
viewed by patients and health professionals. Quality of Life Research: An
International Journal of Quality of Life Aspects of Treatment, Care and
Rehabilitation, 24(10), 2423–2432.
Koopman, R. J., Petroski, G. F., Canfield, S. M., Stuppy, J. A., & Mehr, D. R. (2014).
Development of the PRE-HIT instrument: Patient readiness to engage in health
information technology. BMC Family Practice, 15(1): 18.
Kruskal, J. B. (1964). Multidimensional scaling by optimizing goodness of fit to a
 540 O. Norgaard et al. (2015)

nonmetric hypothesis. Psychometrika, 29(1), 1–27.

Lluch, M. (2011). Healthcare professionals’ organisational barriers to health information
technologies - A literature review. International Journal of Medical Informatics,
80(12), 849–862.
Neter, E., & Brainin, E. (2012). eHealth literacy: Extending the digital divide to the realm
of health information. Journal of Medical Internet Research, 14(1): e19.
Norman, C. (2011). eHealth literacy 2.0: Problems and opportunities with an evolving
concept. Journal of Medical Internet Research, 13(4), e125.
Norman, C. D., & Skinner, H. A. (2006). eHealth literacy: Essential skills for consumer
health in a networked world. Journal of Medical Internet Research, 8(2): e9.
Nutbeam, D. (1998). Health promotion glossary. Health Promotion International, 13(4),
349–364.
Oh, H., Rizo, C., Enkin, M., & Jadad, A. (2005). What is eHealth (3): A systematic
review of published definitions. Journal of Medical Internet Research, 7(1): e1.
Osborne, R. H., Batterham, R. W., Elsworth, G. R., Hawkins, M., & Buchbinder, R.
(2013). The grounded psychometric development and initial validation of the Health
Literacy Questionnaire (HLQ). BMC Public Health, 13(1): 658.
Osborne, R. H., Elsworth, G. R., & Whitfield, K. (2007). The health education impact
questionnaire (heiQ): An outcomes and evaluation measure for patient education and
self-management interventions for people with chronic conditions. Patient Education
and Counseling, 66(2), 192–201.
Roehrer, E., Bjørnes, C. D., Cummings, E., & Nøhr, C. (2014). Human factors
considerations when developing eHealth solutions to support patients : Comparison of
Danish and Australian experiences. International Journal of Healthcare Technology
and Management, 14(3), 157–175.
Trochim, W. M. K. (1987). The concept system (c). New York: Concept Systems
Software Training.
Trochim, W. M. K. (1989a). An introduction to concept mapping for planning and
evaluation. Evaluation and Program Planning, 12(1), 1–16.
Trochim, W. M. K. (1989b). Concept mapping: Soft science or hard art? Evaluation and
Program Planning, 12(1), 87–110.
Ward Jr., J. H. (1963). Hierarchical grouping to optimize an objective function. Journal
of the American Statistical Association, 58(301), 236–244.
World Health Organization. (2015). Health topics: eHealth. Retrieved from
http://www.who.int/topics/ehealth/en/

View publication stats