biocybernetics and biomedicalengineering 41 (2021) 419– 444

Available at www.sciencedirect.com

ScienceDirect

journal homepage: www.elsevier.com/locate/bbe

Review Article

A comparative analysis of automatic classification

and grading methods for knee osteoarthritis
focussing on X-ray images

Deepak Saini a,*, Trilok Chand a,1, Devendra K. Chouhan b, Mahesh Prakash c
a
Punjab Engineering College(Deemed to be University), Sector-12, Chandigarh, India
b
Department of Orthopaedics, Post Graduate Institute of Medical Education & Research (PGIMER), Chandigarh, India
c
Department of Radiology, Post Graduate Institute of Medical Education & Research (PGIMER), Chandigarh, India

A R T I C L E I N F O

Article history: Objective: The purpose of present review paper is to introduce the reader to key directions
Received 29 August 2020 of manual, semi-automatic and automatic knee osteoarthritis (OA) severity classification
Received in revised form from plain radiographs. This is a narrative review article in which we have described recent
2 March 2021 developments in severity evaluation of knee OA from X-ray images. We have primarily
Accepted 3 March 2021 focussed on automatic analysis and have reviewed articles in which machine learning,
Available online 1 April 2021 transfer learning, active learning, etc. have been employed on X-ray images to access
and classify the severity of knee OA.
Methods: All original research articles on OA detection and classification using X-ray
Keywords:
images published in English were searched on PubMed database, Google Scholar, RSNA
Knee osteoarthritis
radiology databases in year 2019. The search terms of ‘‘knee Osteoarthritis” were combined
Convolution neural networks
with search terms ‘‘Machine Learning”, ‘severity” and ‘‘X-ray”.
Deep learning
Results: The initial search on various publication databases revealed a total of 743 results,
Machine learning
out of which only 26 articles were considered relevant to radiographic knee OA severity
Computer aided diagnosis
analysis. The majority of the articles were based on automatic analysis. Manual
segmentation based articles were least in numbers.
Conclusion: Computer aided methods to diagnose knee OA are great tools to detect OA at
ealry stages. Advancements in Human Computer Interface systems have led the
researchers to bridge the gap between machine learning algorithms and expert healthcare
professionals to provide better and timely treatment options to the knee OA affected
patients.
Ó 2021 Nalecz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy
of Sciences. Published by Elsevier B.V. All rights reserved.

* Corresponding author.
E-mail addresses: [email protected] (D. Saini), [email protected] (T. Chand), [email protected]
(D.K. Chouhan), [email protected] (M. Prakash).
1
Sr. Member IEEE.
https://doi.org/10.1016/j.bbe.2021.03.002
0168-8227/Ó 2021 Nalecz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences. Published by Elsevier
B.V. All rights reserved.
420 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420
2. Osteoarthritis, cause, symptoms, diagnosis and machine learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
3. Literature search methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
3.1. Literature search approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
3.2. Exclusion & Inclusion Criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
3.3. Assesed Outcomes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
4. Survey for the available datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
4.1. Public datasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
4.2. Local datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
4.3. Datasets used in corresponding studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
5. Machine learning and deep learning in a nutshell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
5.1. Machine learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
5.1.1. Logistic Regression (LR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
5.1.2. Random Forest (RF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
5.1.3. Linear mixed model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
5.1.4. Naive Bayes classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
5.2. Deep learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
5.2.1. AlexNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
5.2.2. BVLC CaffeNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
5.2.3. VGG-Net group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
5.2.4. ResNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
5.2.5. U-Net . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
5.2.6. ResNeXt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
5.2.7. SENets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
5.2.8. Siamese networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
6. Knee image segmentation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
7. Classification/Assessment of Knee OA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
8. Research directions and open challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
9. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
Funding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Compliance with ethical standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Ethical approval and informed consent:. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Declaration of Competing Interest. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441

1. Introduction

Osteoarthritis (OA) also named as ‘‘Degenerative Joint Dis- inability to diagnose the symptoms at an early stage as there
ease” or ‘‘Wear and Tear arthritis”, is the most common mus- might be possibility of reducing its impact or slow down its
culoskeletal disorder, which mainly affects weight wearing progression of future disability [6]. So, the only available
joints like hip, knee, spine, feet and fingers [1]. Age, heredity, options for sustaining the healthy life is an early diagnosis
injury, hormone disorder, repeated trauma to joint, uric acid and behavioural interventions [7] because at the advanced
or diabetes are many of the few reasons that causes knee stage of OA, joint replacement surgery remains the only alter-
OA [2]. Generally, knee OA occurs in old age due to wear of native. Medical Imaging has been employed for the early diag-
protective tissue between joints (cartilage), but Knee OA nosis of OA [8]. It is being successfully applied to many
may affect younger people due to joint injury or repetitive applications like diagnosis, monitoring and even treating
joint stress from overuse [3]. According to a survey, medical conditions. The advancements in computer hard-
osteoarthritis is the 11th highest global disability factor that ware, software and medical imaging techniques, had syner-
affected 303 million people globally in 2017 [4]. Treatment of gistically led to a rapid rise in the potential use of Artificial
OA costs approximately around 19,000 dollars per year, and Intelligence (AI) in various radiological imaging tasks such
thus it proves to be a major economic burden for world in as prognosis, diagnosis, risk assessment, detection and ther-
today’s date [5]. apy response [9]. Medical Imaging creates the visual represen-
As per the literature, major portion of the treatment cost tation of the interior parts of the body. It also aides in the
arises due to the lack of awareness among patients and establishment of the database of normal anatomy and phys-
 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x 421

Fig. 1 – Stages of Knee OA.

iology which helps to identify possible abnormalities and also ticularly Deep Learning (DL), a subfield of AI makes the com-
serves as source of medical data for further research and puter system capable and intelligent enough that they
study. It includes various biological imaging techniques automatically extract features from images, process or learn
[10,9] like: Magnetic Resonance Imaging(MRI), Computed from those features and at last provide the end-user (human
Tomography(CT), and X-ray. beings) with classification results, known as end-to-end classi-
Out of the many available medical imaging techniques like fication architecture. The inclusion of much-sophisticated soft-
MRI [11,1,12], CT scan [13,14], Ultrasound [15,16], plain radio- ware has revolutionized the field of CADx. The use of better
graph (x-rays) to detect early symptoms of knee OA, x-rays computational resources has led the researchers to explore
have been proven to be least expensive, reliable, readily avail- other imaging modalities like MRI [23–26,11,27] and Ultrasound
able and less hazardous imaging technique. This is the reason (US) [28] to analyze the knee joint suffering from osteoarthritis.
that radiographic imaging is still considered as gold standard Nowadays,with help of AI, researchers can work on multi-
[17] for clinical assessment of bone and joints [18,19]. model and multi-dimensional data to not only classify but to
Physicians usually inspect the X-ray images of OA predict the progression of knee OA [29]. The aim of this article
infected/damaged knees and then classify the severity of is to provide a narrative review of the most relevant articles on
knee OA based on KL grades. KL grading system is the gold detection and classification of knee OA and its severity from X-
standard for grading severity of knee OA and have been ray images. We have also tried to shed light on benefits and lim-
accepted globally for knee OA grading. KL grade splits knee itations of increasing computational power in the knee OA seg-
OA severity into 5 grades from grade 0 to grade 4 as shown mentation and classification. We belive that this analysis could
in Fig. 1 [20,21]. The diagnostic accuracy varies and it relies beofgreat helpforfuture researchersworking in thefieldofknee
merely on physician’s experience and carefulness [22]. For OA severity classification using X-ray images.
the successive grade classification of radiographic knee OA
severity, fine grained image classification is required. But
the classification task is very challenging if using traditional
hand-crafted features derived from texture, pixel, edge and 2. Osteoarthritis, cause, symptoms, diagnosis
object statistics, transforms, histograms, etc. Manually and machine learning
designed feature extraction requires expert domain knowl-
OA is a long-term chronic condition that occurs when the pro-
edge, takes a lot of effort, and is a laborious task. Therefore,
tective tissue between the joints known as the cartilage begins
instead of using manually engineered features, the process
to wear down over time. Due to this thinning of cartilage, bones
of feature extraction has been automated [21].
start rubbing against each other which causes stiffness,
For high-level image processing, automatic feature learning
impaired movement, and pain. OA persists as the body is no
is used for the effective representation of features learned by
longer able to repair the tissues of the joints regularly. Cartilage
transforming raw data inputs (images). So, there exist many
actually cushions the bone ends allowing the movement of the
powerful models in Computer-Aided Diagnosis (CADx) that
joints smoothly and easily. Due to this, there is bone inflamma-
can support clinician’s evaluation and have the power to reach
tion and formation of bone limps in the joints leading to
human level performance. Convolutional Neural Networks
impaired movement of joints, pain, and stiffness. OA can be
(CNN) have been widely used in medical imaging, classification,
caused due to aging, hereditary or due to secondary issues like
image detection, and segmentation as it automaticallylearns all
obesity, injury, hormone disorders, repeated trauma to joint,
the effective and relevant image features [5]. Artificial intelli-
uric acid or diabetes, etc [30–33] Fig. 2 diagrammatically depicts
gence (AI) is a broader term often followed by machine learning
the various causes for Osteoarthritis.
(ML) and deep learning (DL). AI enables computers to mimic
Arthritis generally falls under two categories:
human intelligence and gives a computer the ability to solve
complex problems. The history of the term AI dates back to
(a) Primary:- With age, the water content found in carti-
1950 when AI was termed as human intelligence exhibited by
lage begins to decrease, therefore weakening it which
machines. In 21st century, rapid technological advancement
results in more susceptible to its degradation and less resi-
and availability of larger data sets have led AI to tackle many
lient. This type of OA is caused by aging or due to genetic
computer vision problems like segmentation, classification
problems [34].
and has also openeda new era of computer-aided diagnosis. Par-
422 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

Fig. 2 – Osteoarthritis is directly related to age and obesity. 1lb increase in body weight exherts 3 to 4lb of extra pressure on
knee joints. Persons who over-utilize their joints such as those related to sports, are at more risk of knee OA. Females tends
to have higher probability to get affected by knee OA. Workplace hazards and frequent joint trauma are few of the leading
causes of Knee OA. [2].

(b) Secondary:- This type of OA is not related to age or ment of bone spurs in the joint area [35]. As discussed earlier,
genes. It may show up in early age due to special problems there exist various imaging techniques like MRI [36–38], CT,
like diabetes and obesity, result of injury, athletics or and ultrasound, X-ray is still considered as the effortless,
patients of rheumatoid arthritis, excessive squatting or cheap, easily accessible, and gold standard for the prelimi-
kneeling [34].Fig. 3 summarizes types of Osteoarthritis. nary diagnosis method [9]. The radiographic film is visually
examined by physicians to split it into one of the five KL
Osteoarthritis Symptoms often develop slowly and worsen grades based on many of the pathological features. However,
over time. Pain and stiffness are the most prominent symp- the KL grading system suffers from the subjectivity of the
toms, but many patients suffering from Knee OA have practitioner and its accuracy relies on the physician’s experi-
reported other symptoms like Bone Spurs, Tenderness in ence and the end grade or diagnosis may get affected by inter/
joints, swelling near the joint area, Grating Sensation, and intra rater agreement. Even the same physician may some-
loss of flexibility or limited range of motion [13]. In the dam- times misclassify the severity of the same Knee joint over dif-
aged knee there is a noticeable reduction in space between ferent times. Near grade (grade 3 and grade 4)
the bones forming a joint, also known as Joint Space width misclassification is another major limitation of KL grade
Narrowing (JSN). approach [22]. Osteoarthritis Research Society International
The major pathological features are also visible in plain (OARSI) more recently have proposed a new grading approach
radiographs as the extent of degradation of cartilage can be that is more feature specific and works on simple radio-
measured by visualizing the joint space width and develop- graphs. In this new approach the features like femoral osteo-
phytes (FO), tibial osteophytes (TO), and joint space narrowing
(JSN) are graded separately in compartment wise manner as
shown in Fig. 4 [7].
However, like the KL grading system, this recently pro-
posed OARSI grading system also suffers from human subjec-
tivity, inter/intra rater agreement, and hence all these factors
make early knee OA diagnosis challenging and thus affecting
millions of people worldwide. To provide a common ground
for physicians and doctors around the world, computer-
Fig. 3 – Knee Osteoarthritis types.
 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x 423

Fig. 4 – Examples of knee osteoarthritis features graded according to the Osteoarthritis Research Society (OARSI) grading atlas
and Kellgren-Lawrence (KL) grading scale. [7].

aided diagnosis (CADx) could be used to grade the severity of shown that Deep Learning, particularly convolutional neural
knee OA. Due to the high prevalence of Knee OA, a fully auto- networks (CNNs), has shown groundbreaking results in many
matic knee OA severity grading system is urgently required. CADx [39] and image recognition tasks [5]. CNN constitutes a
Numerous techniques are being proposed by researchers that class of feed-forward networks made up of neurons with
automatically classify the severity of knee OA based on learnable weights and biases, having multiple layers.
pathological features of the knee joint. Potentially
computer-aided methods based on machine learning and 3. Literature search methodology
deep learning could be employed successfully to grade the
severity of knee osteoarthritis and eliminate the inter/intra 3.1. Literature search approach
rater agreement factor from classification. Also, the diagnos-
tic accuracy of these methods already reaches human levels To explore the various research studies focussing on auto-
and even could outperform human experts soon [22]. Deep matic classification and grading methods for knee
learning (DL) is a state-of-the-art Machine learning method osteoarthritis using X-ray images, various publication data-
that learns on the features of the images to detect and accu- bases have been accessed such as Google Scholar, PubMed,
rately classify the grade (KL) of knee OA [7]. Experiments have Medline, and RSNA Radiology. To list down the research arti-

Fig. 5 – Taxonomy of the proposed study.

424 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

cles for the current review, the keywords used were ‘‘knee 4.2. Local datasets
Osteoarthritis, Machine Learning, X-ray, and severity”. All
the articles in which the mentioned keywords appear either Table 2 gives description about the datasets which were cre-
in the title or in the abstract were selected. This initial search ated or collected by the researchers on their own. The
led to a total of 743 publications. researchers leveraged the freedom to collect the knee images
as per their requirements like in few researches the authors
3.2. Exclusion & Inclusion Criteria have acquired plain AP radiographs while some have used
semi-flexed method, some have acquired AnteroPosterio
To short list the number of publications for current review (AP) weight wearing knee radiographs while some without
article, we screened out articles not published in English lan- weight wearing.
guage, excluded the articles that haven’t used 2-d X-ray
images. We included the full-length articles that have foc- 4.3. Datasets used in corresponding studies
cused on X-ray image analysis for knee OA severity grading
using:
different segmentation methods based upon manual, Table 3 gives a berief summary about the various studies that
semi-automatic and automatic approaches.
Various have worked upon different datasets as discussed in subSec-
machine learning and end-to-end architecture based deep tion 4.1 (public) and 4.2 (local). From Table 3 it can be analysed
learning based models. that OAI is the most commonly used dataset while ROAD and
BLSA are least explored datasets on which future studies can
3.3. Assesed Outcomes focus on for more robust and generalized deduction about
knee OA progression and knee OA severity grading. Symbolic
The above search resulted in a total of 26 research articles out representation for the Table 3 has been explained in Table 4
of 743 initial articles. These 26 articles thus obtained are fur- where each symbol signifies the segmentation technique
ther studied in three categories based on segmentation meth- being employed.
ods ‘Manual’, ‘Semi-automatic’ and ‘Automatic’. There are 18
studies based upon public datasets- ‘OAI dataset (13)’, ‘MOST 5. Machine learning and deep learning in a
dataset (3)’, ‘BLSA (2)’, and 10 studies are based on ‘Other’ nutshell
dataset. Apart from this, 70 titles and 7 URL links focussing
on the introduction and basic concepts for the present review The goal of this section is to provide readers a overview of
are added. Overall we have 81 journal articles, 14 conference machine learning (ML) and deep learning (DL) concepts. We
proceedings, 1 Ph.D. thesis, and 7 weblinks (URLs), altogether have briefly described the basic concepts of ML and DL and
combined making 103 references in total. In this review, we have also shed some light on techniques and architectures
have tried to provide the readers a glimpse of the current of various ML/DL based algorithms, used either for segmenta-
state of the art for machine learning based automatic classi- tion or classification, from papers surveyed in this review.
fication of severity of knee OA from X-ray images. The litera-
ture search has been pictorially represented in Fig. 5. 5.1. Machine learning

4. Survey for the available datasets Machine Learning is an intersection of various sub-fields ‘sta-
tistical’, ‘probabilistic’, ‘computer science’, and ‘algorithmic’,
The different research applications/publications have worked making it a capable tool to understand the hidden insights
on either public datasets or have themselves created their significant for developing intelligent applications. Machine
own datasets. The major public datasets available are learning plays a central role in various domains ranging from
Osteoarthritis Initiative (OAI), Multicenter Osteoarthritis data mining, computer vision, Natural Language Processing,
Study (MOST), and Baltimore Longitudinal Study of Aging and designing expert decision-making systems [66]. Algo-
(BLSA). The most common datasets on which researchers rithms in machine learning develop a model based on train-
have extensively worked upon is OAI dataset or OAI derived ing data to make a decision or make predictions. Machine
dataset like KOACAD. So, based on the surveyed literature learning is a subset of artificial intelligence that aims at devel-
the datasets have been reviewed under two categories, viz. oping models capable of making decisions with no/minimum
publicly available datasets and other datasets. Other datasets human intervention. Machine learning gives smart solutions
comprise of the studies in which the authors or researchers in medical healthcare thereby enhancing the accuracy of clin-
have created their own datasets. ical diagnosis and thus improving the healthcare treatment of
the patients. There are a variety of machine learning algo-
4.1. Public datasets rithms ranging in complexity from low (Support vector
machines, Logistic regression, and so on) to high (Neural Net-
Table 1 describes the various publicly available datasets work and ensemble of models). Some of the most widely used
including - the name with which that dataset is available algorithms are as follows:
online, their brief description along with their weblink or ref-
erences. From literature surveyed it has been found out that 5.1.1. Logistic Regression (LR)
OAI dataset is the one on which most of the studies are based Logistic regression is a type of parametric classification model
upon. which is used where the response variable is categorical type.
 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x 425

Table 1 – Public datasets along with their description and URL links.
Dataset Name Dataset Description Number of Weblink/Reference
Participants

OAI 4,796 https://nda.nih.gov/oai [40]

Multi-center, ten year
observational study.

Men and Women
age = 45–79 years.
MOST 3,026 https://most.ucsf.edu/[41]

Lateral knee assessment
for 84 months.

Men, Women average
age = 50–79 years.
BLSA more https://www.nia.nih.gov/research/labs/blsa [42,43]

Is a longitudinal normative than 3,200
aging study started in 1958.mm?>

X-ray images were obtained
in all participants,
irrespective of symptoms
or functional limitations

The basic idea behind Logistic regression is to find a relation- X = design matrix for the fixed effects coefficients b,
ship between features and probability of particular outcome. Z = design matrix of the random forest coefficient b, and.
In LR a sigmoid function is used to map predictions to e = vector of random errors.
probabilities.
The general equation which states the LR model is as Assumption: Random effects and errors are independent of each
follows: other and both multivariate normally distributed.

eðb0 þb1 XÞ
pðXÞ ¼  ð1Þ 5.1.4. Naive Bayes classifier
1 þ eðb0 þb1 XÞ
Naive Bayes is a type of probabilistic classifier based on
where p(X) is the predicted output, b0 is the intercept term famous Baye’s theorem. It is best suited for larger datasets
and b1 is the coefficient for the single input value (x). which may contain millions of images or data samples.
Advantages: It involves simple, fast and easy prediction crite-
5.1.2. Random Forest (RF) ria. It performs well on binary as well as on multi-class clas-
Random forest is a type of supervised machine learning algo- sification. The applications of naı̈ve bayes algorithm includes
rithm which can be used for both classification as well as – Multi-class prediction; text classification including Senti-
regression. The main application of RF is for classification ment analysis; spam filtering, recommendation systems.
purposes. It combines the predictive ability of multiple tree- Bay’s rule is applied to a set of individual variables to form
based models. Random forest classification is an ensemble Naı̈ve Baye’s.
type of classification in which not only one but many classi- Mathematically, Bayes theorem can be written in the fol-
fiers are used. RF creates many decision trees on data sam- lowing manner:
ples, gets prediction from each tree and then finally selects
PðXjYÞ  PðYÞ
the best possible solution/prediction by means of majority PðYjXÞ ¼ ð3Þ
PðXÞ
voting. The main advantage of the RF algorithm is that it
reduces the variance of single tree models and also eliminates Here ’Y’ is the target variable or class output and ’X’ is the
the problem of correlated predictors [67,68]. Fig. 6 shows a set of dependent variables X = X1, X2,- - -, Xn.
canonical diagram of random forest model. Since Naı̈ve classifier assumes the condition of indepen-
dence among the feature variables, therefore the Eq. 3 for
5.1.3. Linear mixed model bayes rule is applied to the set of independent variables to
Linear mixed models also known as ‘multilevel or hierarchical get the output probabilities of particular class given X as:
models’, are a type of regression model which takes into
ðProbability of outcome=evidenceÞ
account both fixed and random effects. They are particularly
¼ ðProbability of likelihood of evidence
used when there is non-independence in the data (as in a case
where we have patient level data along with knee level data  PriorÞ=ðProbability of evidenceÞ
for knee osteoarthritis severity analysis). The Linear mixed Mathematically,
model can be formulated as follow:
PðYjX1 ; X2 ;     ; Xn Þ
y ¼ Xb þ Zb þ e ð2Þ PðX1 jYÞPðX2 jYÞ      PðXn jYÞ  PðYÞ
¼ ð4Þ
PðX1 ÞPðX2 Þ      PðXn Þ
where:
426 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

Table 2 – Other datasets along with their description.

Reference Dataset Description

Marijnissen et al.,2008 [44] 20 healthy and 55 OA affected knee standard radiographs taken according to semi-flexed
method by Buckland-wright and American College of Rheumatology criteria.
Podsiadlo et al.,2008 [45] Plain AP radiographs of left and right knee of 86 subjects. Total number of X-ray images
were 172.
Yoshimura et al., 2010 [46] Total 3040 participants forming three cohort from; Urban, Maountainous and Seacost areas.
Database includes anteroposterior and lateral radiographs of bilateral knees. Research on
Osteoarthritis/ Osteoporosis Against Disability (ROAD) study.
Woloszynski et al., 2010 [47] AP radiograph of human tibia head.
Subramoniam & Rajini, 2013 [48] 50 samples out of which 15 were with normal joint space and 35 were with abnormal joint
space.
Hirvasniemi et al., 2014 [49] AP weight wearing radiographs from both knees of 103 subjects resulting in total of 203
samples.
Subramoniam et al., 2015 [50] 130 digital X-ray images of knee OA symptomatic patients while 10 images of healthy
subjects.
Gornale et al., 2016 [51] 200 knee X-ray images collected from different hospitals and diagnostic centers.
Yoo et al., 2016 [31] Fifth Korean National Health and Nutrition Examination Survey (http://knhanes.cdc.go.kr/
knhanes). Total participants = 2665.
Liu et al., 2020 [30] The data was collected from various hospitals, resulting in 1385 X-ray images.
Saleem et al., 2020 [19] AP X-ray images of 82 subjects (58 bilateral and 24 unilateral).

Posterior probabilities for each class is calculated and class

with maximum value is the final predicted class (result) of the
Regularization method used was ‘Dropout’ which proved
model. very effective in avoiding overfitting.

Overlapping pooling led to reduction in errors by 0.4% and
5.2. Deep learning 0.3% for Top-1 and Top-3 error rates.

Efficacious implementation of Multi-GPU for convolution
Deep learning uses artificial neural networks (ANN) which tasks.
are designed to imitate the way human brain thinks and
ReLU was used as activation function instead of Tanh in
learn. They consist of many layers of neurons stacked onto order to avoid Vanishing Gradient problem and also, the
each other, connected with each other. The layers include use of ReLu leads to faster training.
an input layer, an output layer and numerous hidden layers
Data augmentation was incorporated within the architec-
placed between them. In deep learning the input data is fed ture, one was mirroring and second method was by alter-
into input layer, the data then passes through different lay- ing the intensity.
ers of the network, with each network hierarchically defin-
ing specific features of data, and after finding the
appropriate identifiers for classification the output layer 5.2.2. BVLC CaffeNet
comes out with the result. Although the deep learning After the introduction of AlexNet in 2014, a single GPU version
models gives excellent performance, they have certain lim- of AlexNet known as ’CaffeNet’ is developed by Berkeley
itations. The limitations are; need of larger datasets for bet- Vision and Learning Center (BVLC) in 2014 [70]. The advantage
ter results, expensive hardware like multi-core CPUs, GPUs, of caffeNet over AlexNet is that no data augmentation and
specific deep learning algorithms can only be used for pooling layers were placed before local response normaliza-
specific problems, the output of deep learning algorithms tion layers, as a result caffenet became computationally more
is not comprehended without the use of classifiers. This efficient than AlexNet. In current version of caffeNet, the
part of the section provides a brief introduction and defini- order of pooling layers and local normalization layers is same
tion of various deep learning algorithms that are imple- as that of AlexNet, i.e. pooling layers are placed after local
mented in papers surveyed here. normalization layers. The modified version of caffe Net can
process over 60 million images per day with a single NVIDIA
5.2.1. AlexNet K40 GPU. The online repository can be accessed at ‘ https://
AlexNet was released in the year 2012 [69]. This was the first github.com/BVLC/caffe’ [71].
deep convolutional neural network which outperformed the
classical image object recognition procedures. The network 5.2.3. VGG-Net group
pushed the researchers to make efforts to find out the scope VGG was released in the year 2014 by Visual Geometry Group,
of deep learning in image-based classification problems. Oxford University. The VGG is more deeper network than
The architecture of AlexNet consists of 8 layers out of which AlexNet and has the advantage of smaller convolution filters
5 are convolutional layers and 3 are fully connected layers. of 3 x 3. This was the first network with more than ever depth
The network has 650000 neurons and 60 million trainable (up to 16 or 19 weight layers) [72]. The smaller size of convo-
parameters. Some of the notable features of AlexNet architec- lution filters let the researchers incorporate more weight lay-
ture are listed below: ers.The number of fully connected layers is 3 and the number
 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x 427

Table 3 – Research Publications and the corresponding datasets.

Studies DATASETS
OAI MOST ROAD BLSA Local

Oka et al.,2008 [52] – – – –

Marijnissen et al.,2008 [44] – – – –
Podsiadlo et al.,2008 [45] – – – –
Shamir et al., 2008 [53] – – – –
Shamir et al., 2009 [54] – – – –
Woloszynski et al., 2010 [47] – – – –
Anifah et al., 2011 [55] – – – –
Subramoniam & Rajini, 2013 [48] – – – –
Hirvasniemi et al., 2014 [49] – – – –
Subramoniam et al., 2015 [50] – – – –
Thompson et al., 2015 [56] – – – –
Gornale et al., 2016 [51] – – – –
Yoo et al., 2016 [31] – – –
Antony et al., 2016 [57] – – – –
Antony et al., 2017 [58] – – – –
Suresha et al., 2018 [59] – – – –
Tiulpin et al., 2018 [5] – – –
Norman et al., 2019 [60] – – – –
Brahim et al., 2019 a [61] – – – –
Brahim et al., 2019 b [32] – – – –
Guan et al., 2019 [62] – – – –
Liu et al., 2020 [30] – – – –
Tuilpin et al., 2020 [63] – – – –
Thomas et al., 2020 [64] – – –
Saleem et al., 2020 [19] – – – –
Leung et al., 2020 [65] – – – –

of weight layers is in accordance to the architecture, like VGG-

16 has 13 convolution layers and 3 fully connected layers. Table 4 – Color coding scheme.
Simillarly, VGG-19 has 16 convolution layers and 3 fully con- Color Coding
nected layers. ReLU is the activation function used. Due to Symbol Segmentation Type
more number of weight layers the VGGNet is slow to train
and the network is heavy, over 550 MB and so it has long infer- Manual
Semi-automatic
ence time.
Automatic

5.2.4. ResNet
With introduction of deep convolutional neural network
(DCNN) in 2012, the depth of neural networks has increased
so for, but increase in number of layers does not always guar- Multiple version of ResNet were developed like ResNet-50,
antees increase in accuracy. It has been found that after cer- ResNet-101, ResNet-152. It was also found that ResNet with
tain maximum threshold for number of layers in neural 50/101/152 layers have less error for image classification task
network, the error percentage increases. The problem ’in- in comparison to a 34 layer plain Net [73].
crease in error with increase in number of layers’ is known ResNet is formed by stacking individual block of 2 to 3 con-
as vanishing/exploding gradiens [73]. To overcome this prob- volution layers, known as Residual Blocks. Fig. 7 shows canon-
lem, a new architecture was developed in 2015 which imple- ical form of residual block.
mented the concept of ’Skip-connections’. Skip-connection
skips training from a few layers and connect directly to the 5.2.5. U-Net
output. Other notable features are as follow: U-Net is a class of CNN in which the dense and pooling layers
were replaced transposed 2-d convolution (upsampling) lay-
ers. The upsampling layers unlike dense layers, maintain

The concept of skip-connections led to even more deeper the structural integrity of the image and hence reduce the dis-
architecture and greater improvement in accuracy of the tortion enormously and improves the resolution of output
model. feature map. The U-Net was originlly developed to perform

First CNN based architecture to have feature of ’batch- segmentation of biomedical images. The usp of U-Net is that
normalization’. it works with very few training images and yields pricise seg-

Increased training speed. mentation [74]. U-Net consists of three sections; a. Contrac-

The network has 26 million parameters. tion, b. Bottleneck, and c. Expansion. The main idea behind
428 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

Fig. 6 – Random Forest model.

Siamese network works in way that it takes input through

U-Net is that it uses same feature maps that are used in con- all (identical) subnets, compares the outputs from subnets
traction to expand a vector to a segmented image. and then decides to which class does the input data (image)
belongs to [79]. The siamese network have similar constitu-
5.2.6. ResNeXt tion of convolutional layers, pooling layers but here we don’t
This network was introduced as an improvement to ResNet have softmax layer,but incontrast to CNN, here the difference
and was designed specificaly for image classification tasks. of output from Dense layers is passed thorugh single neuron
ResNeXt architecture uses the concept of ‘cordinality’, which with ’monotonic’ (mostly sigmoid) activation function (0 or 1),
reduces the number of parameters while number of layers for beginners guide to understand the working of Siamese
can be increased. The optimal value of cordinality was found Network, refer to [80]. Fig. 9 shows what a siamese network
out to be 32 [75]. architecture is like.
ResNeXt architecture incorporates parallel not sequential
stacking of layers. It follows ‘split-transform-merge’ stratergy, 6. Knee image segmentation methods
derived from inception model [76]. The image features are
trasposed from downsampling path to upsampling path by This section presents the comprehensive literature review of
skip-connection. ResNext comprises of many ResNext blocks the various methods, techniques and technologies used to
stacked in parallel fashion which enables the blocks to share detect, diagnose and grade the knee OA. Knee OA classifica-
hyperparameters as in case of VGG/ResNet. tion process is achieved in two steps, first being the localiza-
tion (extraction of Region of Interest (ROI) that is extraction of
5.2.7. SENets knee joint area) and second step is to assess the ROI to quan-
The basic building block of SEnets are ‘squeeze and excita- tify the severity of knee OA under various (KL) grades.
tion’ blocks that tends to improve channel interdependences Knee joint area can be extracted by using one of the three
without increasing the computational cost [77]. methods: Manual, Semi-automatic and Automatic methods.
Feature map of each channel is squeezed to single In first two methods human intervention is required at all
numeric value and resulting a vector of size ‘n’ (‘n’ is the or at some place, while in later case human intervention is
number of convolution channels). The single numeric value not required and the computer-based system automatically
is then fed to a two-layer neural network which outputs a vec- performs the RIO extraction and classification.
tor of same size i.e., n. These n values can now be used as
weights on the original feature map and it then scales each A. In manual image segmentation methods, trained
channel based on its importance [77]. Fig. 8 shows basic dif- physicians and expert radiologists visually analyse the
ference between a ResNet block and a SE-ResNet block. joint and carefully map the structure of the joint employ-
ing CAD based methods, simple vision-based tools and
5.2.8. Siamese networks image processing, see Fig. 10.
Siamese networks are those neural networks which contains
Tibia head was localized in the X-ray image by devel-
two or more symmetrical(identical) subnets. The important oping a dissimilarity measure (SDM2) which measures
feature for a network to be called Siamese is that the weights the distance between the texture of images to segment
have to be shared between the identical subnets [5,78]. The the knee joint [47]. In SDM, a sum of earth mover’s dis-
 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x 429

Fig. 7 – Canonical form of Residual Block. Fig. 8 – Comparative visualization of simple ResNet block
and SE-ResNet block.
tance (EMDs) is used to quantify dissimilarities
between textures. assess radiographic features of knee OA. A new interac-

Total of six ROIs were manually placed into the tive tool KIDA (Knee Images Digital Analysis) was pro-
image.Two ROIs were placed into the subchondral bone posed, which extracts the above features as
plate, two immediately below the subchondral bone continuous variables. But the system still requires
plate in subchondral trabecular bone in tibia,and two expert interventions for quantitative evaluations and
in the medial and lateral condyles of femur [49]. The measurements [44].
advantages of manual segmentation are that it is reli-
Minimum joint space width (mJSW) between the
able as the entire process is done under the supervision edges of the tibial plateau and femoral condyle was
of experts, simple to conduct and also no expensive considered as the main radiographic feature to assess
setup or tools are required to perform manual segmen- the progression of knee OA [82]. The X-ray images are
tation. Manual Segmentation is widely accepted as first digitized after which the centre of joint is cropped
ground truth above all the other segmentation tech- manually before being fed to software. Triangle-rule
niques but it has certain limitations too, like it is very based algorithm automatically measure the mJSW by
time consuming and laborious task specially when marking the contours on the edges of tibia and femur
the data set is huge. Since this type of segmentation bone [82].
is completely dependent on the experties of the radiol-
Later Active contour image segmentation was used as
ogist/clinician, there are greater chances of inter and the initial state. Chan-Vese and edge methods are used
intraobserver variability and subjectivity [47,49,81]. and region between the tibia and femur is segmented
B. Semi-automatic methods are also known as interactive for further evaluation [51]. The main benefit of semi-
method in which certain steps of contour extraction are automatic segmentation method is that it provides flex-
automated followed by manual checking/inspection of ibility along with improvement in quality of annotation
the segments and sometimes even editing the segment after manual interventions and incorporation of
contours or boundaries, see Fig. 11. advanced computer vision tools combined with expert

An interactive system was developed [52] which knowledge. This type of segmentation has few disad-
obtains the rough contour of tibia and femur bones vantages also like; whenever required or necessary a
using Roberts filter. The system uses six parameters human expert’s intervention is needed for plotting or
such as joint space area (JSA) at medial and lateral refining the boundary lines of the segmented image,
sides, osteophyte formation, minimum joint space which in return may results in variability due to human
width (mJSW) at medial and lateral sides and last is error. Inter observer and intra observer variations in
tibiofemoral angle (TFA) on anteroposterior X-ray. After measurements of various features happens which
study it was concluded that JSN is only relevant feature makes the results of segmentation ’non-reproducible’
correlated with grading system [52]. [44,51,52,82].

In another study area of osteophyte, subchondral C.; Automatic extraction of the region of interest (ROI) or
bone density, height of tibial eminence is included to automatic image segmentation is a technique in which
430 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

Fig. 9 – Block representation of Siamese network.

Fig. 10 – Manual Segmentation pipeline.

trained algorithms are used to segment an image and is using active shape models [45]. Same model can be
becoming an essential part of clinical decision support used to extract knee joint area. Their system managed
systems and computer-aided diagnosis [83]. Automatic to achieve the similarity index (SI) of 0.83 for medial
methods of segmentation are undoubtedly fast, accurate, and 0.81 for the lateral knee bone regions where the
and comparatively precise and at the same time beneficial SI index is being compared with manually annotated
in clinical trials and pathology. Multiple attempts have ROIs from the professional radiologists [45]. After the
been made till date to localize the knee joint automatically pre-processing and masking of joint area canny edge
without any human interventions and at par with human detection was used to detect bone edges [84].
accuracy.
Another template matching technique was proposed

A fully automatic segmentation method has been [53] to find the ROI (knee joint area) along with a sliding
proposed for extraction of tibial trabecular bone area window strategy. Euclidean distances for every patch
 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x 431

Fig. 11 – Semi-Automatic segmentation pipeline.

of segmentation, so this technique is the fastest of all

the other segmentation techniques and also it produces
accurate/consistent results. Fig. 12 shows a pipeline
structure of automatic segmentation process. We can
process very large datasets in a matter of minutes.
Despite many advantages, there are many disadvan-
tages pertaining to fully automatic segmentation and
they are as follow:

Fig. 12 – Automatic Segmentation pipeline. a. Automatic segmentation is a type of type of unsu-

pervised segmentation technique where Artificial Neu-
ral Network based segmentation algorithms are first
are calculated using sliding window and compared with trained on sample segmented images and are then
predefined knee joint centre template. The window used in practical applications. So, these algorithm
with smallest Euclidean distance is considered to be require extensive training and their performance is
the knee joint area [53]. sensitive to training parameters and gets adversely

Later template matching was combined with Con- affected due to presence of noise/artificats [85].
trast - Limited Adaptive Histogram Equalization b. Active Shape models and Template matching based
(CLAHE) and was able to achieve accuracy of around segmentation models are not suitable for very large
80 percent to 100 percent in detecting knee joint auto- datasets. These models are subjective are not scalable.
matically [55]. The performance of these models depends entirely on

Linear SVM along with the Sobel horizontal image the selection of correct template and accurate identifi-
gradient was used for feature extraction. Along with cation of the landmark [45,53,55,54].
SVM, sobel edge detection is used which considers ver- c. SVM based methods were not able to produce satis-
tical and horizontal image gradients [57]. factory localisation [57].

Significant improvement was acheived in automatic d. FCN based segmentation methods are prone to
localizing of the knee joint area after using Support errors due to variations in the resolution and size,
Vector Machines (SVM). The working models work in and anatomical knee variations [58].
two steps: first is anatomically-based region proposal, e. Automatic segmentation methods are complex to be
and then selecting the best region by evaluating using implemented.
Histogram of Oriented Gradients (HOG) and pre trained Table 5 shows the comparative analysis of various tech-
SVM classifier scores [83]. niques and approaches used in literature review to segment

A fully convolutional neural network (FCN) based knee joint area.
approach was put forward which automatically detects
knee joint area from raw X-ray image [58]. 7. Classification/Assessment of Knee OA

For ROI extraction ”Region proposal neural network”
was used. This model is trained on hand annotated Grading of knee OA from plain radiograph depends upon
knee regions as ground truth. They used the knee loca- pathological features like osteophyte formation (bone spurs),
liser repository of [83] for extraction of knee joints and subchondral sclerosis (hardening of bone) and joint space
does further pre-processing of the data as per their narrowing (JSN). All of these pathological features are inte-
requirements [59]. In Automatic Segmentation, no grated implicitly in scoring system like KL grading system
human interference is present or needed at any step and the OARSI reading which provides features grading dis-
 432
Table 5 – Comparative Analysis of Segmentation methods and techniques.
Mode Study Techniques Advantages Disadvantages Human Interventions & Errors Variability
Subjectivity of the expert

Manual Woloszynski Done manually by expert
Relaible.
Time Consuming. Maximum/ High Incidental Error Interobserver and
et al., 2010 [47] radiologists and physicians
Used as ground truth for the
Experience Based. Intraobserver
with the use of CADx tools analysis of various automatic method.
Subjective. Variability is High [86–88].
and techniques.
Laborious.
Hirvasniemi
et al., 2014 [49]
Semiautomatic Marijnissen KIDA software is proposed
Allows for the incorporation of
Always need expert intervention Moderate Incidental Error Aims at reducing inter-
et al., 2008 [44] for the measurement of CADx tools and human for plotting various lines. and intraobserver varability,
various knee features by the intervention at the same time.
Inter and Intra observer variations but interobserver variability
experts. for the measurement of various features. will still be present [86–88].

Results are not reproducible.

Chance of human error.

diabetes research and clinical practice

Problem of oversight.

Inter observer and inter user difference.
Oka et al., 2008 Roberts filter and vertical
[52] neighbourhood differences.
Subramoniam Local Binary Pattern as
& Rajini., 2013 features and K-Nearest
[48] Neighbour is used for
classification (Euclidean
distance).
Subramoniam Haralick features are
et al., 2015 [50] extacted from ROI, then
SVM is uded for
classification.
Gornale et al., Manual cropping followed
2016 [51] by active contour
segmentation. Chan-Vese
and Edge methods to
segment image.
Brahim et al., Manual marking the tibial
2019 a [61] spines and the lateral and
medial extremities of the
tibia, then algorithm
approximates tibial edge to

4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4
find medial, middle and
lateral ROIs.
Brahim et al., Manual marking of tibial
2019 b [32] spines and the lateral and
medial extremities of the
tibia.Verticle adjustments
of ROI to avoid subchondral
bone sclerosis.
(continued on next page)
Table 5 – (continued)
Mode Study Techniques Advantages Disadvantages Human Interventions & Errors Variability
Subjectivity of the expert

Automatic Podsiadlo Active shape models and morphological
Automatic.
Active shape models Low Systematic Error None (The segmentation
et al., 2008 [45] operations
Fast. are not suitable algorithm trained on specific

Accurate. for big database dataset, so they are application

Beneficial. and do not generalise. specific and may not

No human interference.
Template generalize well.) Also,

Can process large datasets. matching is an the model so developed will
ad hoc method have same bias as the
and can’t be used observer [81].
for large datasets. It
is subjective and
not scalable. Moreover

diabetes research and clinical practice

it depends entirely on
the selection of the template
set and intensity level differences.

SVM based methods are not able
to produce satisfactory localisation accuracy.

FCN are prone to errors do
the variations in the resolution and
sizes and anatomical knee variations.
Shamir et al., Template matching followed by calculation
2008 [53] of Euclidean distance and fisher scores using
sliding window.
Shamir et al.,
2009 [54]
Anifah et al., Template matching along with CLAHE.
2011 [55]
Thomson RFCLM1 was trained to detect the position of
et al., 2015 [56] bones and to locate a set of key landmarkes
across tibia and femur.
Antony et al., Linear SVM along with Sobel edge detection.
2016 [57]
Tiuplin et al., HOG and SVM classifier.
2017 [83]
Antony et al., ”FCN with the help of masking followed by

4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x
2017 [58] bounding box Generation.”
Tiulpin et al., Deep Siamese CNN architecture.
2018 [5]
Suresha et al., ”Region proposal deep neural network.”
2018 [59]
Yoo et al., 2016 Independent predictors like age, sex, BMI,
[31] knee pain, educational status, hypertension,
and moderate activity were used to built the
scoring system and ANN.
Norman et al., U-Net model was used to localize the knee
2019 [60] joint.
Liu et al., 2020 RPN2 with Non-maximal Suppression is used
[30] to find the correct ROI.
Tiulpin et al., random forest regression[90] voting model is
2020 [63] used to localize the knee joint.
Thomas et al., 169-layer convolutional neural network with
2020 [64] a dense convolutional network architecture
was used.
Saleem et al., HOG3 based template matching was used to
2020 [19] automatically localize the knee joint.
Leung et al., Multi-task ResNet with 34 layers was used.
2020 [65]
1
Random Forest Regression Voting Constrained Local Model [89]
2
Region Proposal Network
3
Histogram of oriented gradients

433
434 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

Table 6 – Various classification methods surveyed.

Classifiers Usage
Pre-Processing/Segmentation Classification

Logistic Regression [52][31][62][61]

Logistic Model [52]
Variants of Neural Network [45,91,48]
SVM [57] [47][50][31][57]
Linear Mixed Model [49]
Random Forest [56] [51,56,32]
Neural Network [59][5] [59]
Deep Learning [58] [5][58][64]
Ensemble [60,63]
Manual [19]

tinctly. KL that is Kellgren & Lawrence (KL) grading system ple weighted Nearest Neighbor Rule was enhanced and a
quantifies the knee OA severity under five grades or cate- new method for classification was proposed which
gories [20]. The classification assessment of knee severity is employs the weighted neighbor distances using a com-
approached using two ways in the literature: 1) classification pound hierarchy of algorithms representing morphology
based upon quantification of distinct pathological features (WND-CHARM) algorithm. At first various transforms are
and 2) classification based upon composite grading system applied on raw pixels as well as on the transforms of
like KL grading. transforms to extract set of image features which are fur-
ther accepted or rejected based on the calculated fisher
A. Classification based upon distinct feature quantification:- score. Out of the extracted features, relevant and informa-
Individual parameters/features like JSN at lateral and tive features are further used for classification purposes
medial sides, joint angulation and osteophytes as inputs [54]. An automatic classification system based on Local
were used to classify the severity of knee OA in proposed Binary Pattern (LBP) was proposed. The system first
system KOACAD [52]. New and upgraded system (KIDA) extracts feature and based on extracted features it classi-
was proposed, which considers subchondral bone density, fies the images in two categories normal and abnormal.
tibial eminence height along with other features. The indi- For classification the system uses k-Nearest neighbor clas-
vidual features were assessed as continuous variables and sifier [48]. Haralick features of knee radiographs were used
according to the evaluation significant difference can be for ROI extraction and then kernel function was used for
found between normal and damaged knee. Results evalu- feature extraction.The extarcted features are then classi-
ated are compared with KL grades and significant correla- fied using SVM (Support Vector Machines) [50]. Random
tion is found between them [44]. To monitor the Forest classifier was used to standardise the automatic
development of knee severity, a trainable-rule based algo- classification of Knee OA. Firstly features extracted from
rithm is proposed in [82] which basically focus on mini- image texture of tibia and bone shape are combined and
mum joint space width (mJSW). The normal and weighted sum of two outputs of the classifier was used fur-
damaged knee are classified using joint space width as ther [56]. Improved version of ANN was proposed and Self -
the main metric [84]. JSW calculated is compared with Assessment Scoring system was developed. The prediction
the standard JSW value set. Active shape model and fractal models include ROC curves and selected cut-off points[31].
analysis of bone textures were used in detecting OA [45]. Prediction of KL grades was considered as regression prob-
Automatic approach based on distance based Active Shape lem using a continuous distance based mean squared
Models is proposed in [91] which calculates the geometric error as a metric. Deep convolutional neural networks like
parameters between femur bone and tibia bone. Various VGG16, BVLC Caffenet, and VGG 128 which are pre-trained
features like first four moments, texture analysis features, on ImageNet and fine tune on own dataset are used for
haralick, shape and statistical features were computed to classification [57]. Classification and regression losses
assist the system to accomplish classification task. These were minimized by employing 5-layer CNN which uses
features are classified by Random Forest Classifier [51]. multi-objective convolutional learning for the optimiza-
B. Classification based upon composite grading:-An automatic tion of weighted ratio of categorical cross-entropy and
detection of OA using KL grades was proposed in which mean-squared error [58]. Object classification neural net-
image analysis was performed by identifying some image work were used in [59]. A deep neural network model
content descriptors and applying various image trans- was pre-trained on ImageNet for knee classification. The
forms. For rejecting the noisy features and selecting the network is trained using the KL graded dataset which is
most informative ones, content descriptors are assigned initially graded by expert raters and radiologists. New clas-
weights using Fisher scores. Hence simple weighted near- sification approach whose architecture is based on the
est neighbor rule was used to classify the resulting feature Deep Siamese CNN was proposed in [5]. Network is trained
vector and predicts the KL grade to X-ray image [53]. Sim- to learn the symmetry in the images. Network input is
Table 7 – Comparative Analysis of methods for Knee Osteoarthritis classification
Source Classification type Dataset used Methodology Assessment

Oka et al., 2008 [52] Quantitative analysis ROAD study: 1979 knee x-rays out of total 2002 images of 1001 KOACAD system quantifies the knee KOACAD system so developed measures
subjects (AP View). based on various knee parameters. six major parameters within one
Correlation among parameters, KL and second.
OARSI system is done Spearman’s and
Pearson correlation test. Multivariate
logistic regression analysis is done at
the end.
fv Marijnissen et al., 2008 [44] Quantitative analysis 20 Healthy knees X-ray images; 55 OA infected knees x-rays KIDA an interactive tool is developed Correlation between KIDA parameter
(acquired using semi-flexed method). which takes around 10 min to analyze data and KL grade were calculated.
each radiograph. Bone density, JSW,

diabetes research and clinical practice

osteophytes area, joint angle and tibial
height are measured as continuous
variable and results are correlated with
KL grades.
Podsiadlo et al., 2008 [45] Quantitative analysis Independent Dataset: Total X-ray images = 172; ROI selected Active shape model is used for the
Automatic ROI selection.
images = 132; Images for training = 40. delineation of the bone plates which is
Calculated Fractal Dimensions (FDs)
further followed by fractal analysis for and Fractal Signature (FS) and it is
the assessment of knee OA. suggested that change in FD to off-
set wass less that 0.035 as com-
pared to 0.05.
Shamir et al., 2008 [53] Automatic Baltimore Longitudinal Study of Aging (BLSA) [42] Image transforms and content KL3: 91.5%. KL 2: 80.4%. KL 1: 57% (with
descriptors that are relevant are respect to normal knee)
selected using image analysis. Further
they are assigned with features weights
using fisher scores. At the end,
classification is done by simple
weighted nearest neighbor rule under
various KL grades.
Shamir et al., 2009 [54] Automatic Baltimore Longitudinal Study of Aging (BLSA) [42] Various transforms are applied on the KL3: 72%, KL2: 62% (with respect to
features and on the set of relevant normal knee)
features selected, classification
algorithm is applied. Weighted neighbor
distances using a compound hierarchy
of algorithms representing morphology

4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x
(WND-CHARM) algorithm is used for
classification. It is KNN variant.
Woloszynski et al., 2010 [47] Qualitative Local dataset comprising of 68 healthy knee x-rays and 69 OA Two SDM (signature dissimilarity Accuracy: 78.8% with SDM-NN and
affected knee x-rays. measure) based methods are used for 85.4% with SDM-SVM SDM based
knee detection; SDM-NN and SDM-SVM. system outperformed WND-CHARM
SVM is used with Radial Basis kernel classifier sysyem.
and validation technique is leave-one-
out cross validation.
Subramoniam & Rajini, 2013 [48] Qualitative Total of 50 X-ray images were acquired out of which 15 were It classifies under 2 categories that is
Accuarcy in case of Manhatton and
normal knee X-ray images while 35 were abnormal cases. normal or OA knee. Local Binary Pattern correlation distance measure =
(LBP) and kNearest neighbour classifier 97.37%.
are used in this system.
Accuarcy in case of Euclidean dis-
tance measure = 96.75%.
Subramoniam et al., 2015 [50] Qualitative Total of 130 knee X-ray images (30 normal and 100 abnormal) After the extraction of region of interest 99% accuracy in diagnosing skeletal
using haralick features, image is disorder caused by OA.
classifies using SVM.
Thomson et al., 2015 [56] Automatic 500 knee radiographs from OAI initiative dataset. Two different classifiers are used here to Accuracy improved to 84.9% from 78.9%
analyze the texture of the image. Simple in case of automated detection.
weighted sum of the output of the two
random forest classifiers is used further.
Gornale et al., 2016 [51] Quantitative analysis 200 knee x-rays were acquired from different hospitals. After the ROI segmentation, different Accuracy of about 87.92 % is achieved.
types of features like haralick, shape,
statistical, texture analysis etc. is
evaluated. This list of features is fed to
Random Forest Classifier for
classification task.
(continued on next page)

435
 436
Table 7 – (continued)
Source Classification type Dataset used Methodology Assessment

Yoo et al., 2016 [31] Automatic KNHANES V-1 in 2010: 2665OAI study: 4731 MLP neural network with back Accuarcy:-
propagation algorithm is implemented.
73% for knee OA detection.

88% for symptomatic knee OA.
Antony et al., 2016 [57] Automatic OAI study: 4,476 After the joint area segmentation, Multi-class classification accuracy =
various pretrained deep neural 59.6%
networks like VGG16, BVLC Caffenet,
and VGG 128 are used for classification.
These pre-trained models are fine-
tuned are their own dataset.
Antony et al., 2017 [58] Automatic OAI study: 4,476 MOST study: 3,026 Convolutional neural network (CNN) Multiclass accuracy: 60.3 %

diabetes research and clinical practice

containing 5 layers is trained from
scratch to minimise the classification
and regression losses. CNN is jointly
trained for classification as well as
regression.
Suresha et al., 2018 [59] Automatic OAI study: 7549 Object classification neural network KL 0: 78.9% KL 1: 63.2% KL 2: 66.7% KL
which is a deep neural network model is 3:79% KL 4:87%
pre-trained on ImageNet for knee
severity classification.
Tiulpin et al., 2018 [5] Automatic OAI study: 4,796 MOST study: 3,026 Classification approach whose Multiclass accuracy: 66.71%
architecture is based on the Deep
Siamese CNN is proposed. Network is
trained to learn the symmetry in the
images. Network input is explicitly
mapped with relevant attention zone so
that only that region is considered
specifically during decision making.
Norman et al., 2019 [60] Automatic OAI study: 4,490 bilateral fixed-flexion knee radiographs
Knee joints were localised using U-
U-Net localised Knee joints with
Net model. 98.3% accuracy.

Ensemble of DenseNets is employed
Specificity and Senstivity values
for classification of knee OA during testing for No, Mild, Moder-
severity. ate and Severe OA are as follows:

Saliency maps further validated the 83.7%, 70.2%, 68.9%, 86.0% and

4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4
results. 86.1%, 83.8%,97.1%, 99.1%
Brahim et al., 2019 a [61] Automatic OAI public dataset: 688 knee radiographs with K-L grade 0 and K-L
Semi-automatic segmentation to
Combined PSD features for orienta-
grade 2 were taken. extract medial ROI. tions 0° and 90° with ICA.

Power spectral density (PSD) as fea-
using logistic regression classifier:
ture for classification. Accuracy = 78.924%, sensitivity =

Independent component analysis 79.651% and Specificity = 78.198%.
(ICA) is applied on combined PSDs
of 0° and 90° orientations.
Brahim et al., 2019 b [32] Automatic: Random Forest and Naive OAI public dataset: 514 knee radiographs for both K-L grade 0 and
Semi-automatic segmentation to
Intensity normalization using
Bayes. K-L grade 2 were taken. extract medial ROI. multivariate linear regression reduced

First 10 discriminant components intersubject variability and increased
from ICA as features for separation between K-L grade images.
classification. Accuracy = 82.98  2.12% Senstivity =

Qualitative Assessment by Kull- 87.15  4.25% Specificity = 80.65 
back-Liebler and Jeffreys Divergence 1.42%.
matrics.
Guan et al., 2019 [62] Automatic: Deep Learning models (VGG- OAI dataset: 600 subjects with K-L grade 1,2,3 grouped as pre-ROA
Combination of DL models and clinical Accuracy = 83.2% Senstivity = 80%
19 & DenseNet) combined with clinical and grade 4 as ROA of knee. data is used to predict the ROA Specificity = 78%
data progression.
Liu et al., 2020 [30] Automatic Private dataset of 2105 individuals Total of 2770 X-ray images Combination of Region Proposal Accuracy = 74.3%Senstivity = 93.6%
Network (RPN) and Fast R-CNN called Specificity = 74.2%
Faster R-CNN. RPN trained to localize
knee joint in raw X-ray image and Fast
R-CNN to classify the severity of Knee
OA (using Focal loss and larger anchors).
(continued on next page)
 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x 437

explicitly mapped with relevant attention zone so that

Cohen’s Kappa = 0.82, and balanced

Cohen’s Kappa = 0.90, and balanced

accuracy = 80.66% for JSN in medial

AUC of 0.87 at 95% confidence interval.

only that region is considered only during decision

DL based multitask model has greater

Avg. Precision = 0.70 Avg. Recall = 0.69

Accuracy = 97.14%. F1 score = 98.40%.

compartment for OARSI grading.
accuracy = 66.68% for K-L grade.

Avg. F1 score = 0.70 Cohen weighted

making.

In Table 6 it has been surveyed that various classifiers such

Cohen’s Kappa = 0.8507.

as Support Vector Machines (SVM), Randon Forest (RF), Neural
Network, deep learning architectures have been used by the
researchers for pre-processing or segmentation purposes.
Kappa = 0.86
Assessment

The classifiers are generally used to classify the objects in dif-

ferent categories or classes, as in our case the classifiers can
be used to classify the knee joints in different severity grades

as per KL grading scheme, but the classifiers can also be used

to extract the required region of interest (knee joint area)
with network weights from a model

combined with clinical risk factors

169-layer convolutional network

Random data augmentation is used.

Backpropagation derived Saliency
map used to qualitatively under-

validation, is used to predict the

to predict Knee OA and probability

ResNet-34, trained with transfer
learning and 7-fold nested cross-

Output of multitask DL model was

HOG feature based template matching.
A DL based method to simultaneously

residual networks viz. SE-Resnet-50 &

Knee Joint area(ROI) is localised using

from raw X-ray image of the knee, or sometimes to normalize

predict K-L and OARSI grades using

calculated JSW is compared with

transfer learning. Used two deep

the image.
To predict the presence of OA,
stand the trained model.
pretrained on Imagenet.

It can be concluded from Table 6 that there is a great scope

standard normal knee JSW.

to use machine learning based methods for pre-processing of

image-based TKR1.

knee radiographs apart from being used for classification pur-

SE-Resnet-50-32x4d.

poses alone.
Methodology

Table 7 shows the comparative analysis of various tech-

of TKR.

niques and methods used for classification of knee OA sever-

ity. From Table 6 and Table 7 it can be deduced that computer
aided diagnosis have almost prevailed the field of image
based diagnostics of underlying deformities. With advance-
140 knee X-ray images of 82 subjects. Bilateral radiographs of 58

patients (No TKR1). Mean age = 64  8 years.WOMAC pain score2

Women: 58%, mean age = 60.9 years and Men: 42%, mean age

patients (After TKR1) and 364 (142 men and 222 women) control

ments in computational power, the use of machines have

Total of 728 subjects from OAI. 364 (142 men and 222 women)

not been limited to be a support system for humans but have

OAI dataset of 4796 participants and MOST dataset of 3026

came at a level where machines can even surpass the

participants. X-rays are acuired with beam angle of 10°

human’s diagnostic power. Advanced machine learning algo-

subjects and unilateral radiographs of 24 subjects.

rithms can process huge quantity of data, simple text based

32116 single limb images of 3606 patients.

or complex image/video based data to get insights. From the

literature it can be concluded that ML and DL based algo-
rithms can look inside the complex information and help or
assist the doctors in decision making. With boom in develop-
ment of deep learning based architectures, automatic knee
OA classification by machines can be used for clinical pur-
Knee-related Quality of Life from Knee Injury and Osteoarthritis Outcome Score

poses but in supervision of human expert.

& KOOS QoL3 [65]
= 61.3 years.
Dataset used

OAI dataset:

8. Research directions and open challenges

Western Ontario and McMaster Universities Osteoarthritis Index.

The easiest way to diagnose any type of OA is to analyse the

radiographs for any abnormality or deformity in bone struc-

ture like narrowing of joint space in knee or formation of bone
Automatic: Multiple-task Deep Learning

spurs, but the process of manual inspection of the radio-

graphs is very time consuming, difficult, cumbersome and
depends heavily on the expertise of human evaluator. Very
often even the expert radiologists fail to notice a slightest
change in bone structure, a noticing symptom of onset of
Classification type

OA, and hence unable to diagnose the disease at early stage.

Computer aided diagnoses of radiographic knee OA have
Automatic

Automatic

Automatic

emerged as a supporting tool for radiologists and surgeons

model.

to detect the onset of OA at early stages. As stated in the lit-

erature that OA is non curable disease so early diagnosis
Total Knee Replacement

and early treatment are the only solution to reduce the

Table 7 – (continued)

impact on human life. These computers assisted methods

Thomas et al., 2020 [64]

Saleem et al., 2020 [19]

Tuilpin et al., 2020 [63]

Leung et al., 2020 [65]

(semi-automatic and automatic) have proved to be promising,

fast and reliable tool to detect the knee OA. Lot of work has
already been explored on these methods but there still a long
way to go.
Source

3
438 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

The growth of artificial intelligence and deep learning due to lack of inter-organisation collaboration, this data
methods in knee osteoarthritis severity analysis is clear from remains hidden from the scientific community. So, in
the extensive literature covered in above sections discussed. order to utilise the full potential of AI, the scattered data
So far various machine learning and deep learning methods needs to be integrated/assembled. A global vault of data
have been used in this field for the purpose of segmentation can be framed where all the data like ’annotated image,
feature extraction and classification or grading but still there biomarkers, clinical data, patients demographic details’,
is a greater room in order to increase the uasability of related to knee OA can be stored. A new form of collabora-
Machine Learning alias Artificial intelligence based automatic tive learning model called ’federated learning’ has emerged
knee OA severity grading algorithms. Any classification task fast [98], where the machine learning algorithms are
mailnly has two steps; 1. Segmentation and 2. Classification. trained across multiple decentralized edge devices (mobile
Studies has shown that integeration of DL based algorithm phones, hospital computers) holding local data, without
along with traditional model-based methods for segmenta- exchanging them [99].
tion purpose has achieved the best results in term of Dice II. Interpretation: Deep neural network has cashed the
Similarity coefficient (DSC) between 85.8% to 90% [92]. Earlier abundance of image data, but a fully automatic deep learn-
segmentation techniques has to rely on human experts for ing models are self-directed as they rely on complicated
guidance which sometimes results in iter- and intra observer web of neuaral network to produce there results (feature
ambiguity, but now AI powered algorithms have taken over extraction,prediction or classification) thus it makes very
these tradional techniques of segmentation. difficult for researchers to interprate these results.This
AI have significantly enhanced the accuracy and efficiency limitation of tranperance and interpretation is known as
of algorithms to detect and classify knee OA severity grade by ’black-box’ problem of deep neaural networks. Some
many folds and thus have motivated the researchers to researchers have tried to address this problem by using
develop more sophisticated algorithms for automatic knee slaiency maps [64,93] to visualize that how a neural net-
OA severity grading. Scope of deep learning has been work has arrived to a particular conclusion. The methods
extended to a wide range of application ranging from auto- are also being used to visualize that how a deep learning
matic segmentation to automatic detection to automatic clas- neural network sees a image. To increase interpretability
sification of knee OA severity. Now the much of the research and justification of how AI driven algorithms works, a
is being done to predict the onset of knne OA [39] so that this new kind of field of ’Explainable-AI’ has emerged [100].
cronic disease can be contained in its early stage. To accom- III. Quantum computing in medical image analysis: From
plish this multi-model based approaches [93] has been imple- the literature surveyed above, it can be concluded that
mented which make use of radiographic data combined with Machine learning and deep learning have achieved
clinical data. A recent study [94] has shown that gait data and impressive results in classifying and predicting knee OA,
radiographic images are complementary with repsect to knee and all has become possible due to increased computa-
OA classification, and combining the both can outperform the tional power, data availability and algorithmic advances.
traditional/common deep learning based methods that rely However, we have almost reached the physical limits in
on just radiographic image data. terms of speed, where as the amount of data is increasing
Intelligent systems are being developed which identifies and datsets are becoming vast. Given the above chal-
the most relevant features of Knee OA structural progressors lenges, use of Quantun Computing may be used to acceler-
to predict the onset of knee OA at early stages [95]. Another ate the training process of available/existing learning
research [96] concluded that combining inflammatory periph- models to discover hidden patterns within the data [101].
eral blood gene expression with imaging biomarker can Quantum Neural Networks (QNN) [102] have highlighted
enhance the prediction accuracy of radiohraphic progression the use of quantum computing in classification task. The
in knee OA significantly. Upto now the use of modern day future research can make use of this unexplored comput-
intelligent, self learning, highly efficient Artificial intelligence ing power in many medical image based diagnosis prob-
powered knee OA severity classification algorithm seems to lems and may give a boost to a very new, emerging field
be settled but there are many hurdels that need to be ’computational medicine’ [103].
addressed like image based data availability and security,
human bias or inter- and intra observer variability during seg- Some of the remaining research directions and open chal-
mentation, and most challenging is the interpretability of the lenges have been tabulated in Table 8 which may gives a path
classification results. All these challenges altogether gives to readers to explore more opportunities in this area of knee
new researchers to explore the field of medical imaging and OA severity grading and to take the current state of the art
diagnosis. Future challenges and research directions have to next level.
been enumerated on the basis of the following factors:
9. Conclusion
I. Data: The deep learning based algorithms need huge
amount of data in order to make very fair, concrete, and The present study reviews the various methods and chal-
generalized classification, but limited availability of well lenges that could be explored and worked upon so that these
annotated data [97] poses a greater challenge for upcoming automated classification and grading methods for knee OA
research. Although the medical feternity and healthcare can be used for the clinical studies or medical diagnosis. A
providers generate enormous amount of image data, but lot of methods have been explored for pre-processing, feature
Table 8 – Research Challenges and Future Directions for knee OA classification and severity grading focussing on Computer Aided methods.
References Research Challenges and Future Directions

On basis of dataset On basis of pre-processing On basis of classification Other Criteria basis

Oka et al., 2008 [52] To validate the sensitivity of KOACAD To find the correlation between pain and NA
Structural measure of knee joint can be
system, investigation on longitudinal data to Knee OA, periarticular disorders such as bone evaluated to grade the knee OA severity
be performed [52]. marrow edema and spontaneous [52].
osteonecrosis can be included [52].
To understand the association between
knee pain and radiographic features, a
comparison between KOACAD parame-
ters and MRI findings (evaluated over
defined period of time) needs to be done
[52]
Marijnissen et al., 2008 [44] The study can be extended to solve the NA NA
Test–retest evaluations of radiographic

diabetes research and clinical practice

problem of ‘‘Systematic bias” by including procedures should be performed in
more than one observer and to increase the future studies [44].
image dataset [44].
Sensitivity of the system to structural
bone changes needs to be evaluated [44].

A study needs to be done to find whether
KIDA is sensitive enough in measuring
the changes in OA parameters (e.g. angle
deviations and subluxations) in an
acceptable period of time or not [44].
Podsiadlo et al., 2008 [45], The work can be extended to include a
The knee border delineation methods NA
The accuracy of ASM can be improved by
Shamir et al., 2008 [53] greater number of X-ray images [53] along can be used to automate the image pro- constructing the statistical model at dif-
with relevant clinical data [53]to increase the cessing tasks [45]. ferent spatial scales [45].
detection accuracy of the system.
Initial anchor point for ASM can be auto-
Testing of automated method needs to
matically selected for segmentation pur- be done to detect the onset of knee OA
pose [45]. before its radiographic appearance. The
same idea can be extended to study the
hand X-ray images for any evidence of
OA or OA progression [53].
Shamir et al., 2009 [54] NA NA Important clinical outcomes like symptoms There is a greater need for more rigorous
presence or severity of OA to be included for studies to determine when a knee has
studies related to development of transitioned from normal grade to a higher
radiographic OA [54]. grade [54].
Woloszynski et al., 2010 [47] Include larger databases in clinical set.
Training image size can be increased to Multiclassifiers can be used [47].
Th work can be extended to classify other

4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x
60  60 pixels. medical images like Ultrasound (US),

Sharp film screens can be used to reduce Magnetic Rasonance Imaging (MRI), chest
noise. radiographs, dermoscopic images of skin,
and 3-D images [47].
Subramoniam et al., 2015 [50] NA NA NA Through efficient training the Haralick
features and SVM based classification
algorithm can be extended to diagnose other
skeletal disorders [50].

Thomson et al., 2015 [56] NA It will be interesting to investigate the effect A regressor can be trained to quantify the OA The method can be used to analyse to bone
of including more textural information from severity on a continuous scale rather than on remodelling and Osteophytes formation and
X-ray images on accuracies [56]. a discrete scale [56]. to study their effects on OA severity
classification [56].
Gornale et al., 2016 [51] NA NA NA There is a need to develop the automated
methods for finding the more accurate
association between the OA related pain and
clinical symptoms [51].
Yoo et al., 2016 [31] The model developed should accumulate or NA A model can be developed to distinguish The work can be extended to find out how
consider larger prospective data in terms of between tibiofemoral and patellofemoral knee related physical activities pose a direct
more images or clinical data to progressively knee OA [31]. risk for knee OA [31].
classify the knee OA on a continuous scale
[31].
(continued on next page)

439
 440
Table 8 – (continued)
References Research Challenges and Future Directions

On basis of dataset On basis of pre-processing On basis of classification Other Criteria basis

Antony et al., 2016 [57] More number of labelled knee radiographs CNN or region-based CNN can be used to While using pre-trained models there is a An end to end deep learning model can be
can be included to validate the results [57]. localize the knee joint in raw knee X-ray chance of some discrepancies, so model developed which will perform all three steps
image [57]. training from scratch can be done to viz. knee joint area localization, feature
streamline the classification process [57]. extraction and knee OA severity grading in
one go [57].

Antony et al., 2017 [58] NA An end to end network integrating FCN for knee localization and the CNN for classification needs The work can be extended to compare the
to be developed in order to improve the fine-grained classification [58]. classification accuracy of automatic
quantification method to that of human
expert [58].

diabetes research and clinical practice

Suresh et al., 2018 [59] NA NA NA
The work can be extended to make a
performance comparison between deep
learnig models and health-care professionals
using small sample size.

The study can be carried forward by
incorporating structured information
like demographic details of the patients
[59].

Also, the automatic segmentation tech-
niques can be used to segment the knee
joint area in MRI images [59].
Tiulpin et al., 2018 [5] Larger amount of data from multiple datasets Without changing the image resolution, a Different loss function that itself optimizes A model needs to be developed that has been
required for testing and validating so that appropriate data filtering technique can be the Kappa coefficient can be employed [5]. trained on MOST dataset and its
more generalized method can be developed developed tht could improve the results [5]. classification performance to classify the
[5]. images acquired without standardized
settings needs to be done [5].
Norman et al., 2019 [60] NA NA NA The efficiency of the model gets hampered
due to presence of some hardware in the
knees, so another machine learning
algorithm can be developed which can detect
the presence of hardware in the knee images
and can assess the OA progression in
”hardware present” and normal cases [60].
Brahim et al., 2019 a [61] NA NA Logistic regression classifier can be employed NA

4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4
to avoid the overfitting [61].

Brahim et al., 2019 b [32] Calibrated X-ray devices be used. NA NA. Demographic details like age, gender can be
included.

Liu et al., 2020 [30] requires a large amount of training data to To improve the quality of annotation in order In order to reduce the dependence of the In order to improve the performance of the
obtain a well performing model [30]. to improve the accuracy of the model, a model on high quality image dataset, semi- model, image information added with
system can be developed which can combine supervised learning methods can be added DICOM data of the patient can be used for
the segmentation process, thus improve the [30]. analysis, which may further help in
performance of the model [30]. diagnosing the disease [30].
Tuilpin et al., 2020 [63] Localized knee image data obtained from Some additional OARSI features like (medial Computational heaviness of the ensemble Attention maps produced can be analysed to
different patients can be used for model tibial attrition, medial tibia sclerosis and approach can be reduced by employing get a better insight about the decisions made
training [63]. lateral femoral sclerosis) could be added as techniques like knowledge distillation [63]. by CNN [63].
features[63].
Thomas et al., 2020 [64] Larger number of knee images acquired NA NA NA
under diverse environments can be included
to validate the performance [64].
Saleem et al., 2020 [19] NA Automatic system could be developed to NA NA
divide the bilateral radiographs in two parts,
each part containing single knee.
Leung et al., 2020 [65] Image datasets can be extended to include NA NA Deep learning prediction model with 3-D MRI
datasets other than OAI dataset alone [65]. data can be developed either by extending
the 2-D CNN approaches or by developing the
3-D CNN approach directly [65].
 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x 441

extraction and classification tasks, but the results are still far [6] Cross M, Smith E, Hoy D, Nolte S, Ackerman I, Fransen M,
away to be used in practical scenarios. So, a lot of advanced Bridgett L, Williams S, Guillemin F, Hill CL, Laslett LL, Jones
deep learning or ensemble of classifiers can be explored for G, Cicuttini F, Osborne R, Vos T, Buchbinder R, Woolf A,
March L. The global burden of hip and knee osteoarthritis:
better classification accuracies for improved diagnosis of knee
estimates from the global burden of disease 2010 study. Ann
OA. Future studies should investigate how the models per- Rheumatic Diseases 2014;73(7):1323–30.
formed while classifying and predicting radiographic knee [7] Tiulpin A, Saarakkala S, Automatic grading of individual
OA or some other forms of skeletal disorders on image data knee osteoarthritis features in plain radiographs using deep
collected from varying imaging modalities or their combina- convolutional neural networks, arXiv preprint
tions. In some scenarios these classification models serve to arXiv:1907.08020; 2019.
[8] Kawathekar PP, Karande KJ, Severity analysis of
further increase the toolset available to scientists and engi-
osteoarthritis of knee joint from x-ray images: A literature
neers working to both understand and develop Human Com-
review. In :2014 International Conference on Signal
puter Interaction system for diagnosing knee OA to improve propagation and computer technology (ICSPCT 2014), pp.
human lives. 648–652, IEEE, 2014.
In future, we are planning to extend our survey on auto- [9] Kijowski R, Demehri S, Roemer F, Guermazi A. Osteoarthritis
matic knee OA classification and grading methods based on year in review 2019: imaging. Osteoarthritis Cartilage
combination of various imaging modalities like X-ray com- 2020;28(3):285–95.
[10] Giger ML, Machine learning in medical imaging, J Am
bined with MRI dataset or X-ray combined with ultrasound
College Radiology, vol. 15, no. 3, Part B, pp. 512–520, 2018.
dataset. We have also planned to survey the studies in which
Data Science: Big Data Machine Learning and Artificial
ensemble of classification techinques has been employed to Intelligence.
classify the disease. [11] Barr AJ, Dube B, Hensor EM, Kingsbury SR, Peat G, Bowes MA,
Sharples LD, Conaghan PG. The relationship between three-
dimensional knee mri bone shape and total knee
Funding
replacement—a case control study: data from the
osteoarthritis initiative. Rheumatology 2016;55(9):1585–93.
This study has no funding.
[12] Hayashi D, Roemer FW, Guermazi A. Magnetic resonance
imaging assessment of knee osteoarthritis: current and
Compliance with ethical standards developing new concepts and techniques. Clin Exp
Rheumatol 2019;37(120):S88–95.
Ethical approval and informed consent: [13] Hamai S, Dunbar NJ, Moro-oka T-A, Miura H, Iwamoto Y,
Banks SA. Physiological sagittal plane patellar kinematics
during dynamic deep knee flexion. Int Orthopaedics 2013;37
Ethical approval and informed consent were not required for
(8):1477–82.
this study. [14] Holzer L, Kraiger M, Talakic E, Fritz G, Avian A, Hofmeister A,
et al. Microstructural analysis of subchondral bone in knee
Declaration of Competing Interest osteoarthritis. Osteoporosis Int 2020:1–9.
[15] Majidi H, Niksolat F, Anbari K. Comparing the accuracy of
radiography and sonography in detection of knee
The authors declare that they have no known competing
osteoarthritis: A diagnostic study. Open Access Macedonian
financial interests or personal relationships that could have J Med Sci 2019;7(23):4015.
appeared to influence the work reported in this paper. [16] Desai P, Hacihaliloglu I. Knee-cartilage segmentation and
thickness measurement from 2d ultrasound. J Imaging
2019;5(4):43.
[17] Lovrenovic Z, Doumit M. Development and testing of a
R E F E R E N C E S
passive walking assist exoskeleton. Biocybernetics Biomed
Eng 2019;39(4):992–1004.
[18] Aprovitola A, Gallo L. Knee bone segmentation from mri: A
[1] Eijgenraam SM, Chaudhari AS, Reijman M, Bierma-Zeinstra classification and literature review. Biocybernetics Biomed
SM, Hargreaves BA, Runhaar J, Heijboer FW, Gold GE, Oei EH. Eng 2016;36(2):437–49.
Time-saving opportunities in knee osteoarthritis: T 2 [19] Saleem M, Farid MS, Saleem S, Khan MH. X-ray image
mapping and structural imaging of the knee using a single analysis for automated knee osteoarthritis detection. Signal,
5-min mri scan. Eur Radiol 2020;30(4):2231–40. Image Video Processing 2020:1–9.
[2] versusarthritis.org, ”Versus arthritis, 2019, osteoarthritis [20] Kellgren JH, Lawrence JS. Radiological assessment of osteo-
(oa).” 2019. arthrosis. Ann Rheumatic Diseases 1957;16(4):494–502.
[3] Ackerman IN, Kemp JL, Crossley KM, Culvenor AG, Hinman [21] Antony AJ, Automatic quantification of radiographic knee
RS. Hip and knee osteoarthritis affects younger people, too. J osteoarthritis severity and associated diagnostic features
Orthopaedic Sports Phys Therapy 2017;47(2):67–79. using deep convolutional neural networks. PhD thesis,
[4] Safiri S, Kolahi AA, Hoy D, Smith E, Bettampadi D, Dublin City University; 2018..
Mansournia MA, Almasi-Hashiani A, Ashrafi-Asgarabad A, [22] Chen P,Gao L, Shi X, Allen K,Yang L. Fullyautomaticknee
Moradi-Lakeh M, Qorbani M. Global, regional and national osteoarthritisseveritygradingusingdeepneuralnetworkswith
burden of rheumatoid arthritis 1990–2017: a systematic anovelordinal loss.Comput MedImagingGraphics2019;75:06.
analysis of the global burden of disease study 2017. Ann [23] Bien N, Rajpurkar P, Ball RL, Irvin J, Park A, Jones E, Bereket
Rheumatic Diseases 2019;78(11):1463–71. M, Patel BN, Yeom KW, Shpanskaya K, et al. Deep-learning-
[5] Tiulpin A, Thevenot J, Rahtu E, Lehenkari P, Saarakkala S. assisted diagnosis for knee magnetic resonance imaging:
Automatic knee osteoarthritis diagnosis from plain development and retrospective validation of mrnet. PLoS
radiographs: A deep learning-based approach. Sci Rep 2018;8 Med. 2018;15(11) e1002699.
(1):1727.
442 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

[24] Culvenor AG, Øiestad BE, Hart HF, Stefanik JJ, Guermazi A, [42] Shock NW, Normal human aging: The Baltimore
Crossley KM. Prevalence of knee osteoarthritis features on longitudinal study of aging. US Department of Health and
magnetic resonance imaging in asymptomatic uninjured Human Services, Public Health Service, National; 1984..
adults: a systematic review and meta-analysis. British J [43] ‘‘The baltimore longitudinal study of aging — national
Sports Med 2019;53(20):1268–78. institute on aging.” https://www.nia.nih.gov/research/labs/
[25] Nagai K, Nakamura T, Fu FH. The diagnosis of early blsa, 2020. (Accessed on 08/12/2020).
osteoarthritis of the knee using magnetic resonance [44] Marijnissen AC, Vincken KL, Vos PA, Saris D, Viergever M,
imaging. Ann Jt 2018;3:1–5. Bijlsma J, Bartels L, Lafeber F. Knee images digital analysis
[26] Hossain MB, Pingguan-Murphy B, Chai HY, Salim MIM, Dewi (kida): a novel method to quantify individual radiographic
DEO, Supriyanto E, Lai KW. Improved ultrasound imaging for features of knee osteoarthritis in detail. Osteoarthritis
knee osteoarthritis detection. In: Medical Imaging Cartilage 2008;16(2):234–43.
Technology. Springer; 2015. p. 1–40. [45] Podsiadlo P, Wolski M, Stachowiak G. Automated selection of
[27] Saygılı A, Albayrak S. A new computer-based approach for trabecular bone regions in knee radiographs. Med Phys
fully automated segmentation of knee meniscus from 2008;35(5):1870–83.
magnetic resonance images. Biocybernetics Biomed Eng [46] Yoshimura N, Muraki S, Oka H, Kawaguchi H, Nakamura K,
2017;37(3):432–42. Akune T. Cohort profile: research on osteoarthritis/
[28] Rath PD, Pandey SC. Ultrasound findings in knee of patients osteoporosis against disability study. Int J Epidemiol 2010;39
of osteoarthritis and their correlation with pain. Int J Res (4):988–95.
Orthopaedics 2019;5(5):944. [47] Woloszynski T, Podsiadlo P, Stachowiak G, Kurzynski M. A
[29] Cai G, Cicuttini F, Aitken D, Laslett LL, Zhu Z, Winzenberg signature dissimilarity measure for trabecular bone texture
T, Jones G. Comparison of radiographic and mri in knee radiographs. Med Phys 2010;37(5):2030–42.
osteoarthritis definitions and their combination for [48] Subramoniam M, Rajini V, Statistical feature based
prediction of tibial cartilage loss, knee symptoms and classification of arthritis in knee x-ray images using local
total knee replacement: a longitudinal study. binary pattern. In: 2013 International Conference on
Osteoarthritis and Cartilage 2020. Circuits, Power and Computing Technologies (ICCPCT), pp.
[30] Liu B, Luo J, Huang H. Toward automatic quantification of 873–875, IEEE, 2013..
knee osteoarthritis severity using improved faster r-cnn. Int [49] Hirvasniemi J, Thevenot J, Immonen V, Liikavainio T,
J Computer Assisted Radiol Surgery 2020;15(3):457–66. Pulkkinen P, Jämsä T, Arokoski J, Saarakkala S.
[31] Yoo TK, Kim DW, Choi SB, Park JS. Simple scoring system Quantification of differences in bone texture from plain
and artificial neural network for knee osteoarthritis risk radiographs in knees with and without osteoarthritis.
prediction: a cross-sectional study. PloS one 2016;11(2) Osteoarthritis Cartilage 2014;22(10):1724–31.
e0148724. [50] Subramoniam M, Barani S, Rajini M. A non-invasive
[32] Brahim A, Jennane R, Riad R, Janvier T, Khedher L, Toumi H, computer aided diagnosis of osteoarthritis from digital x-ray
Lespessailles E. A decision support tool for early detection of images. Biomed Res 2015;26(4).
knee osteoarthritis using x-ray imaging and machine [51] Gornale SS, Patravali PU, Manza RR. Detection of
learning: Data from the osteoarthritis initiative. osteoarthritis using knee X-ray image analyses: a machine
Computerized Med Imaging Graphics 2019;73:11–8. vision based approach. Int J Computer Appl 2016;1.
[33] Charlesworth J, Fitzpatrick J, Perera NKP, Orchard J. [52] Oka H, Muraki S, Akune T, Mabuchi A, Suzuki T, Yoshida H,
Osteoarthritis-a systematic review of long-term safety Yamamoto S, Nakamura K, Yoshimura N, Kawaguchi H.
implications for osteoarthritis of the knee. BMC Fully automatic quantification of knee osteoarthritis
Musculoskeletal Disorders 2019;20(1):1–12. severity on plain radiographs. Osteoarthritis Cartilage
[34] Lespasio MJ, Piuzzi NS, Husni ME, Muschler GF, Guarino A, 2008;16(11):1300–6.
Mont MA. Knee osteoarthritis: a primer. Permanente J 2017;21. [53] Shamir L, Ling SM, Scott Jr WW, Bos A, Orlov N, Macura TJ,
[35] Hernández-Molina G, Neogi T, Hunter DJ, Niu J, Guermazi A, Eckley DM, Ferrucci L, Goldberg IG. Knee x-ray image
Reichenbach S, Roemer F, McLennan C, Felson DT. The analysis method for automated detection of osteoarthritis.
association of bone attrition with knee pain and other mri IEEE Trans Biomed Eng 2008;56(2):407–15.
features of osteoarthritis. Ann Rheumatic Diseases 2008;67 [54] Shamir L, Ling SM, Scott W, Hochberg M, Ferrucci L,
(1):43–7. Goldberg IG. Early detection of radiographic knee
[36] Raeissadat SA, Ghorbani E, Taheri MS, Soleimani R, Rayegani osteoarthritis using computer-aided analysis. Osteoarthritis
SM, Babaee M, Payami S. Mri changes after platelet rich Cartilage 2009;17(10):1307–12.
plasma injection in knee osteoarthritis (randomized clinical [55] Anifah L, Purnama IKE, Hariadi M, Purnomo MH. Automatic
trial). J Pain Res 2020;13:65. segmentation of impaired joint space area for osteoarthritis
[37] Liu J, Chen L, Tu Y, Chen X, Hu K, Tu Y, Lin M, Xie G, Chen S, knee on X-ray image using gabor filter based morphology
Huang J, et al. Different exercise modalities relieve pain process. IPTEK J Technol Sci 2011;22(3).
syndrome in patients with knee osteoarthritis and [56] Thomson J, O’Neill T, Felson D, Cootes T. Automated shape
modulate the dorsolateral prefrontal cortex: a multiple and texture analysis for detection of osteoarthritis from
mode mri study. Brain, Behavior, Immunity 2019;82:253–63. radiographs of the knee. In: International Conference on
[38] Kiselev J, Ziegler B, Schwalbe H-J, Franke R, Wolf U. Medical Image Computing and Computer-Assisted
Detection of osteoarthritis using acoustic emission analysis. Intervention. Springer; 2015. p. 127–34.
Med Eng Phys 2019;65:57–60. [57] Antony J, McGuinness K, O’Connor NE, Moran K.
[39] Lim J, Kim J, Cheon S. A deep neural network-based method Quantifying radiographic knee osteoarthritis severity using
for early detection of osteoarthritis using statistical data. Int deep convolutional neural networks. In: 2016 23rd
J Environ Res Public Health 2019;16(7):1281. International Conference on Pattern Recognition
[40] ‘‘Oai.” https://nda.nih.gov/oai, 2020. (Accessed on 08/12/ (ICPR). IEEE; 2016. p. 1195–200.
2020). [58] Antony J, McGuinness K, Moran K, O’Connor NE. Automatic
[41] ‘‘Most dataset.” http://most.ucsf.edu/default.asp, 2019. detection of knee joints and quantification of knee
(Accessed on 08/12/2019). osteoarthritis severity using convolutional neural networks.
 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4 x x x 443

In: International conference on machine learning and data convolutions. In: Proceedings of the IEEE conference on
mining in pattern recognition. Springer; 2017. p. 376–90. computer vision and pattern recognition, pp. 1–9, 2015.
[59] Suresha S, Kidziński L, Halilaj E, Gold G, Delp S. Automated [77] Hu J, Shen L, Sun G, Squeeze-and-excitation networks. In:
staging of knee osteoarthritis severity using deep neural Proceedings of the IEEE conference on computer vision and
networks. Osteoarthritis Cartilage 2018;26:S441. pattern recognition, pp. 7132–7141, 2018.
[60] Norman B, Pedoia V, Noworolski A, Link TM, Majumdar S. [78] Chicco D. Siamese neural networks: An overview. Artificial
Applying densely connected convolutional neural networks Neural Networks 2020:73–94.
for staging osteoarthritis severity from plain radiographs. J [79] Li MD, Chang K, Bearce B, Chang CY, Huang AJ, Campbell JP,
Digital Imaging 2019;32(3):471–7. Brown JM, Singh P, Hoebel KV, Erdoğmusß D, et al. Siamese
[61] Brahim A, Riad R, Jennane R, Knee osteoarthritis detection neural networks for continuous disease severity evaluation
using power spectral density: Data from the osteoarthritis and change detection in medical imaging. NPJ Digital Med.
initiative. In: Proceedings of International Conference on 2020;3(1):1–9.
Computer Analysis of Images and Patterns, (Salerno, Italy), [80] ‘‘Siamese networks. line by line explanation for beginners —
pp. 480–487, Springer, 2019.. by krishna prasad — towards data science.”
[62] Guan B, Liu F, Mizaian A, Demhri S, Neogi T, Guermazi A, https://towardsdatascience.com/siamese-networks-line-by-
Kijowski R. Deep learning approach to predict radiographic line-explanation-for-beginners-55b8be1d2fc6. (Accessed on
knee osteoarthritis progression. Osteoarthritis Cartilage 12/18/2020).
2019;27:S395–6. [81] Starmans MP, van der Voort SR, Tovar JMC, Veenland JF,
[63] Tiulpin A, Saarakkala S. Auomatic grading of individual Klein S, Niessen WJ. Radiomics: data mining using
knee osteoarthritis features in plain radiographs using deep quantitative medical image features. In: Handbook of
convolutional neural networks. Osteoarthritis Cartilage Medical Image Computing and Computer Assisted
2020;28:S308. Intervention. Elsevier; 2020. p. 429–56.
[64] Thomas KA, Kidziński Ł, Halilaj E, Fleming SL, Venkataraman [82] Duryea J, Li J, Peterfy C, Gordon C, Genant H. Trainable rule-
GR, Oei EH, Gold GE, Delp SL. Automated classification of based algorithm for the measurement of joint space width
radiographic knee osteoarthritis severity using deep neural in digital radiographic images of the knee. Med Phys 2000;27
networks. Radiol Artificial Intell 2020;2(2) e190065. (3):580–91.
[65] Leung K, Zhang B, Tan J, Shen Y, Geras KJ, Babb JS, Cho K, [83] Tiulpin A, Thevenot J, Rahtu E, Saarakkala S, A novel method
Chang G, Deniz CM. Prediction of total knee replacement for automatic localization of joint area on knee plain
and diagnosis of osteoarthritis by using deep learning on radiographs. In: Scandinavian Conference on Image
knee radiographs: data from the osteoarthritis initiative. Analysis, pp. 290–301, Springer, 2017..
Radiology 2020. p. 192091. [84] Bindushree R, Kubakaddi S, Urs N, Detection of knee
[66] Hügle M, Omoumi P, van Laar JM, Boedecker J, Hügle T. osteoarthritis by measuring the joint space width in knee x
Applied machine learning and artificial intelligence in ray images. Int J Electron Commun, ISSN, pp. 2321–5984,
rheumatology. Rheumatol Adv Practice 2020;4(1). p. rkaa005. 2015..
[67] Abedin J, Antony J, McGuinness K, Moran K, O’Connor NE, [85] Sharma N, Aggarwal LM. Automated medical image
Rebholz-Schuhmann D, Newell J. Predicting knee segmentation techniques. J Med. Phys/Assoc Med Phys India
osteoarthritis severity: comparative modeling based on 2010;35(1):3.
patient’s data and plain x-ray images. Sci Rep 2019;9(1):1–11. [86] Heye T, Merkle EM, Reiner CS, Davenport MS, Horvath JJ,
[68] Breiman L. Random forests. Mach Lsearning 2001;45(1): Feuerlein S, Breault SR, Gall P, Bashir MR, Dale BM, et al.
5–32. Reproducibility of dynamic contrast-enhanced mr imaging.
[69] Krizhevsky A, Sutskever I, Hinton GE. Imagenet part ii. comparison of intra-and interobserver variability
classification with deep convolutional neural networks. with manual region of interest placement versus
Commun ACM 2017;60(6):84–90. semiautomatic lesion segmentation and histogram
[70] Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, analysis. Radiology 2013;266(3):812–21.
Guadarrama S, Darrell T. Caffe: Convolutional architecture [87] Yushkevich PA, Piven J, Hazlett HC, Smith RG, Ho S, Gee JC,
for fast feature embedding. In: Proceedings of the 22nd ACM Gerig G. User-guided 3d active contour segmentation of
international conference on Multimedia. p. 675–8. anatomical structures: significantly improved efficiency and
[71] Github - bvlc/caffe: Caffe: a fast open framework for deep reliability. Neuroimage 2006;31(3):1116–28.
learning. https://github.com/BVLC/caffe. (Accessed on 12/ [88] Parmar C, Velazquez ER, Leijenaar R, Jermoumi M, Carvalho
16/2020). S, Mak RH, Mitra S, Shankar BU, Kikinis R, Haibe-Kains B,
[72] Simonyan K, Zisserman A, Very deep convolutional et al. Robust radiomics feature quantification using
networks for large-scale image recognition, arXiv preprint semiautomatic volumetric segmentation. PloS one 2014;9(7)
arXiv:1409.1556, 2014. e102107.
[73] He K, Zhang X, Ren S, Sun J. Deep residual learning for image [89] Cootes TF, Ionita MC, Lindner C, Sauer P, Robust and
recognition. In: Proceedings of the IEEE conference on accurate shape model fitting using random forest regression
computer vision and pattern recognition. p. 770–8. voting. In: European Conference on Computer Vision, pp.
[74] Ronneberger O, Fischer P, Brox T. U-net: Convolutional 278–291, Springer, 2012..
networks for biomedical image segmentation. In: [90] Lindner C, Thiagarajah S, Wilkinson JM, Wallis GA, Cootes
International Conference on Medical image computing TF, arcOGEN Consortium, et al., ‘‘Fully automatic
and computer-assisted intervention. Springer; 2015. p. segmentation of the proximal femur using random forest
234–41. regression voting,” IEEE transactions on medical imaging,
[75] Xie S, Girshick R, Dollár P, Tu Z, He K. Aggregated residual vol. 32, no. 8, pp. 1462–1472, 2013..
transformations for deep neural networks. In: Proceedings [91] Lee H-C, Lee J-S, Lin MC-J, Wu C-H, Sun Y-N. Automatic
of the IEEE conference on computer vision and pattern assessment of knee osteoarthritis parameters from two-
recognition. p. 1492–500. dimensional x-ray image. First International Conference on
[76] Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Innovative Computing, Information and Control-Volume I
Erhan D, Vanhoucke V, Rabinovich A, Going deeper with (ICICIC’06), vol. 2. IEEE; 2006. p. 673–6.
444 diabetes research and clinical practice 4 1 ( 2 0 2 1 ) 4 1 9 –4 4 4

[92] Ebrahimkhani S, Jaward MH, Cicuttini FM, Dharmaratne A, [97] Antony J, McGuinness K, Moran K, O’Connor NE. Feature
Wang Y, de Herrera AGS, A review on segmentation of knee learning to automatically assess radiographic knee
articular cartilage: from conventional methods towards osteoarthritis severity. In: Deep Learners and Deep Learner
deep learning, Artificial Intelligence in Medicine, p. 101851, Descriptors for Medical Applications. Springer; 2020. p. 9–93.
2020. [98] ‘‘Google ai blog: Federated learning: Collaborative machine
[93] Tiulpin A, Klein S, Bierma-Zeinstra SM, Thevenot J, Rahtu E, learning without centralized training data.” https://
van Meurs J, Oei EH, Saarakkala S. Multimodal machine ai.googleblog.com/2017/04/federated-learning-collaborative.
learning-based knee osteoarthritis progression prediction html. (Accessed on 12/14/2020).
from plain radiographs and clinical data. Sci Rep 2019;9 [99] Li T, Sahu AK, Talwalkar A, Smith V. Federated learning:
(1):1–11. Challenges, methods, and future directions. IEEE Signal
[94] Kwon SB, Han H-S, Lee MC, Kim HC, Ku Y, et al. Machine Processing Magazine 2020;37(3):50–60.
learning-based automatic classification of knee [100] Adadi A, Berrada M. Peeking inside the black-box: A survey
osteoarthritis severity using gait data and radiographic on explainable artificial intelligence (xai). IEEE Access
images. IEEE Access 2020;8:120597–603. 2018;6:52138–60.
[95] Jamshidi A, Leclercq M, Labbe A, Pelletier J-P, Abram F, Droit A, [101] Moustakidis S, Christodoulou E, Papageorgiou E, Kokkotis C,
Martel-Pelletier J, Identification of the most important Papandrianos N, Tsaopoulos D. Application of machine
features of knee osteoarthritis structural progressors using intelligence for osteoarthritis classification: a classical
machine learning methods, Therapeutic advances in implementation and a quantum perspective. Quantum
musculoskeletal disease, vol. 12, p. 1759720X20933468, 2020.. Mach Intell 2019;1(3):73–86.
[96] Attur M, Krasnokutsky S, Zhou H, Samuels J, Chang G, [102] Farhi E, Neven H, Classification with quantum neural
Bencardino J, Rosenthal P, Rybak L, Huebner JL, Kraus VB, networks on near term processors, arXiv preprint
et al. The combination of an inflammatory peripheral blood arXiv:1802.06002, 2018.
gene expression and imaging biomarkers enhance [103] Nielsen MA, Neural networks and deep learning, vol. 2018.
prediction of radiographic progression in knee Determination press San Francisco, CA, 2015.
osteoarthritis. Arthritis Res. Therapy 2020;22(1):1–17.