Computer Science Review 35 (2020) 100203

Contents lists available at ScienceDirect

Computer Science Review

journal homepage: www.elsevier.com/locate/cosrev

Review article

Application of deep learning for retinal image analysis: A review

∗
Maryam Badar a , Muhammad Haris a , , Anam Fatima b
a
School of Electrical Engineering and Computer Science, National University of Sciences and Technology, Islamabad, Pakistan
b
University of Engineering and Technology, Lahore, Pakistan

article info a b s t r a c t

Article history: Retinal image analysis holds an imperative position for the identification and classification of retinal
Received 13 May 2019 diseases such as Diabetic Retinopathy (DR), Age Related Macular Degeneration (AMD), Macular Bunker,
Received in revised form 14 September 2019 Retinoblastoma, Retinal Detachment, and Retinitis Pigmentosa. Automated identification of retinal
Accepted 4 November 2019
diseases is a big step towards early diagnosis and prevention of exacerbation of the disease. A number
Available online xxxx
of state-of-the-art methods have been developed in the past that helped in the automatic segmentation
Keywords: and identification of retinal landmarks and pathologies. However, the current unprecedented advance-
Deep learning ments in deep learning and modern imaging modalities in ophthalmology have opened a whole new
Deep neural network arena for researchers. This paper is a review of deep learning techniques applied to 2-D fundus and 3-D
Convolutional neural network Optical Coherence Tomography (OCT) retinal images for automated classification of retinal landmarks,
Auto-encoder
pathology, and disease classification. The methodologies are analyzed in terms of sensitivity, specificity,
Sparse stacked auto-encoder
Area under ROC curve, accuracy, and F score on publicly available datasets which includes DRIVE,
De-noised sparse auto-encoder
Softmax STARE, CHASE_DB1, DRiDB, NIH AREDS, ARIA, MESSIDOR-2, E-OPTHA, EyePACS-1 DIARETDB and OCT
Random forest image datasets.
Rectified linear unit © 2019 Elsevier Inc. All rights reserved.
Hidden layers

Contents

1. Introduction......................................................................................................................................................................................................................... 3
1.1. Methodology of literature search ........................................................................................................................................................................ 3
1.2. Inclusion criteria .................................................................................................................................................................................................... 3
1.3. Exclusion criteria ................................................................................................................................................................................................... 4
1.4. Selection of papers ................................................................................................................................................................................................ 4
2. Diagnostic analysis and screening of retinal images ..................................................................................................................................................... 4
2.1. Retinal imaging modalities ................................................................................................................................................................................... 4
2.1.1. Fundus photography .............................................................................................................................................................................. 4
2.1.2. 3-D optical coherence tomography (OCT)........................................................................................................................................... 5
2.2. Retinal anatomical structures............................................................................................................................................................................... 5
2.3. Disease manifestations as retinal impairments ................................................................................................................................................. 5
2.3.1. Diabetic retinopathy (DR) ..................................................................................................................................................................... 5
2.3.2. Age related macular degeneration (AMD)........................................................................................................................................... 7
2.3.3. Cardiovascular diseases ......................................................................................................................................................................... 7
2.3.4. Glaucoma................................................................................................................................................................................................. 7
2.4. Quantitative measures for retinal image analysis ............................................................................................................................................. 7
3. Deep-learning...................................................................................................................................................................................................................... 8
3.1. Supervised learning ............................................................................................................................................................................................... 9
3.2. Unsupervised learning........................................................................................................................................................................................... 9
3.3. Challenges to deep learning ................................................................................................................................................................................. 9
3.3.1. Lack of real cognizance behind deep learning techniques ............................................................................................................... 9
3.3.2. Difficult training ..................................................................................................................................................................................... 9
4. Learning methods applied for retinal image analysis.................................................................................................................................................... 9
4.1. Segmentation of retinal vasculature.................................................................................................................................................................... 10

∗ Corresponding author.
E-mail address: [email protected] (M. Haris).

https://doi.org/10.1016/j.cosrev.2019.100203
1574-0137/© 2019 Elsevier Inc. All rights reserved.
2 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

4.1.1. Supervised methods............................................................................................................................................................................... 10

4.1.2. Hybrid methods...................................................................................................................................................................................... 13
4.1.3. Unsupervised methods .......................................................................................................................................................................... 13
4.2. Segmentation of exudates .................................................................................................................................................................................... 13
4.2.1. Supervised methods............................................................................................................................................................................... 13
4.3. Segmentation of microaneurysms ....................................................................................................................................................................... 14
4.3.1. Unsupervised methods .......................................................................................................................................................................... 14
4.4. Simultaneous segmentation of retinal landmarks and pathologies................................................................................................................. 14
4.4.1. Supervised methods............................................................................................................................................................................... 14
4.5. Retinal disease classification ................................................................................................................................................................................ 15
4.5.1. Supervised methods............................................................................................................................................................................... 15
4.5.2. Unsupervised methods .......................................................................................................................................................................... 16
5. Discussion............................................................................................................................................................................................................................ 16
6. Conclusion ........................................................................................................................................................................................................................... 17
Declaration of competing interest.................................................................................................................................................................................... 17
References ........................................................................................................................................................................................................................... 17

We have analyzed and compared algorithms on the basis of a

1. Introduction diverse set of performance metrics which include: sensitivity,
specificity, area under ROC curve, and accuracy. The objective of
Automatic classification of ophthalmologic and cardiovascu- this work is to summarize the current progress made in the field
lar diseases through analysis of retinal images has become an of ophthalmology using deep learning techniques.
established practice in telemedicine. Earlier techniques involved
manual segmentation but it was monotonous, time-consuming, 1.1. Methodology of literature search
inconvenient, labor intensive, observer driven, and required profi-
cient skill [1] whereas computer-aided detection of retinal abnor- A systematic methodology has been developed for searching
malities is cost-effective, feasible, objective, and does not require literature for this study. This systematic methodology includes
efficiently trained clinicians to grade the images [2]. Development the following stages:
of screening systems are helpful in early diagnosis and real time • Defining research problem
classification of retinal diseases such as Diabetic Retinopathy
• Finding pertinent articles that satisfy the pre-set inclusion
(DR), Age Related Macular Degeneration (AMD), Macular Bunker,
criteria
Retinoblastoma, Retinal Detachment, and Retinitis Pigmentosa
• Extracting relevant data from articles
[3]. A number of template, edge, and morphology based algo-
• Assessment of quality of extracted data
rithms have been presented in the past for auto delineation of
retinal landmarks and retinal pathology [4,5]. Moreover, several All the literature that has been reviewed in this survey was
supervised and unsupervised neural networks based methods retrieved by conducting iterative and exhaustive searches on
have also been employed for retinal image analysis. Many super- following databases:
vised methodologies adopted SVM, MLP, ANN and decision trees
[6–8]. Similarly, matched filtering and model based approaches 1. IEEE Xplore Digital Library, (http://ieeexplore.ieee.org)
have been scrutinized for the purpose of unsupervised retinal 2. Springer Link, (http://link.springer.com/)
abnormality detection [9]. 3. ScienceDirect, (http://www.sciencedirect.com/)
All these state-of-the-art methods required manual feature 4. American academy of Ophthalmology, (http://www.aaojou
designing through SURF, SIFT, and HOG feature descriptors [10– rnal.org/)
12]. The explicit domain knowledge is mandatory requirement for 5. The JAMA Network, (http://jamanetwork.com/journals/jam
this kind of hand crafted feature extraction. Most of the times the a)
results obtained through them are more specialized on a dataset 6. Investigative Ophthalmology and Visual Science, (http://
and generalization is not achieved. Recent advancements in visual iovs.arvojournals.org/)
recognition via deep learning have invigorated researchers to
All journal papers as well as conference papers that have
employ these techniques in the field of ophthalmology as well.
been published in the proceeding of above mentioned biblio-
Automatic learning of intricate features in the retinal images
graphic databases have been included in this review. Keywords
can be obtained by the use of deep learning. The supervised
that have been used to perform the search are: deep learn-
and unsupervised multi-layer Deep Neural Networks (DNN) allow
ing, deep neural network (DNN), convolutional neural network
generalized high level feature extraction from raw data image.
(CNN), auto-encoder, stacked sparse auto-encoder (SSAE), de-
Whereas, retinal image analysis based on deep learning has out-
noised sparse auto-encoder (SDAE), softmax, random forest, rec-
performed the traditional methods both for 2-D fundus images
tified linear unit, hidden layers, DR, exudate detection/ segmen-
and 3-D Optical Coherence Tomography (OCT) images.
tation, microaneurysms detection/segmentation, Optic Disc (OD)
Abràmoff et al., have discussed retinal imaging and retinal
localization, fovea localization, retinal blood vessel segmentation,
image analysis based on 2-D fundus and 3-D OCT images. How-
AMD, automated screening, retinal disease classification.
ever, this review encompasses only conventional techniques for
retinal deformity detection. Moreover there are many other re- 1.2. Inclusion criteria
views and survey papers which cover the domains of retinal
landmark detection, retinal pathology segmentation, and retinal ‘‘Application of deep learning for retinal image analysis’’ has
disease classification [1,13]. The major contribution of this paper been set as the cue statement for literature search. Abstracts and
lies in the fact that there is no existing survey paper which titles of all collected articles are checked against the research cue
covers deep learning based retinal image analysis. This work is and only those articles are selected which provide deep learning
a review of 2-D fundus and 3-D OCT retinal images’ analysis based algorithms for segmentation of retinal anatomical struc-
using modern deep learning techniques for automated identifi- tures and retinal disease classification. All the relevant content
cation of retinal landmarks, pathology, and disease classification. up till December, 2018 has been included in this survey.
 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203 3

1.3. Exclusion criteria Moreover, blood vessels and retinal pigments absorb blue light
resulting in a dark background which enhances specular reflec-
The main focus of this survey is to review deep learning tions and scattering in anterior fundus layers. On the other hand,
based techniques for retinal image analysis. Therefore, the articles green light provides the best global retinal view because the
whose algorithms are not based on deep learning concepts are retinal pigmentations reflect green light more than blue light pro-
excluded from this survey. Moreover, the articles from impact viding excellent contrast. Hence, green filter is employed in color
factor journals and reputed conferences have been selected in this fundus photography for improved visualization of retinal vas-
review. We have avoided inclusion of such papers which were culature, drusen, exudates, and hemorrhages. Retinal pigments,
published in local journals and conferences. blood vessels and optic nerve appear almost featureless in red
light and overall contrast of retinal images is reduced. Hence, red
1.4. Selection of papers light is only used for revealing the choroidal pattern, pigmentary
disturbances, choroidal ruptures, choroidal naevi, and choroidal
Articles were selected through vigilant scrutiny of inclusion melanomas [16].
and exclusion criteria. Selected articles were downloaded in pdf Red free photography is the process of retinal image acqui-
format and saved. The nomenclature used for saving selected sition in the presence of an illumination source with a specific
papers depict publishing year, journal name/conference name, color filter to block red light, usually green filter is used for this
publisher, main author name, and title. This type of nomenclature purpose. This kind of fundus photography is effective in view-
helped in indexing the articles and resulted in efficient retrieval ing retinal blood vessels, hemorrhages, pale lesions (exudates
in accordance with queries. Citations of all the articles have also and drusen), epiretinal membranes, and retinal nerve fiber layer
been downloaded and saved in EndNote library. defects [17].
Stereo fundus photography [18] helps in documenting the
2. Diagnostic analysis and screening of retinal images retinal structures because of its increased depth resolution fea-
ture. This type of photography involves simultaneous or sequen-
2.1. Retinal imaging modalities tial visualization of retina through two cameras with different
angles of observation insuring least stress to the patient and it
Precise imaging of retinal tissues and brain tissues is very is viable for diagnosis of macular edema (ME) and sub retinal
important for the diagnosis and treatment of retinal impairments. neovascularization.
Czech scientist Jan Evangelista Purkyně [14] invented principles Hyper-spectral imaging allows efficient visualization of reti-
of ophthalmoscope in 1823 as a first attempt towards direct in- nal lesions with high spatial and spectral resolution. The illumi-
spection of retina. Since then many imaging modalities have been nation source has multiple wavelength bands and their reflection
developed for non-destructive visualization of retinal anatomical is recorded in the form of hyper-spectral image intensities. This
structures. Fundus photography has been found to be effective for type of imaging modality is feasible for retinal blood oxygen
early screening and diagnosis of three most important causes of saturation analysis (oximetry) [19] through likening of spectral
blindness in industrialized world i.e. macular degeneration, glau- absorptions by retinal arteries, veins, and surrounding regions.
coma, and DR. The 2-D representation of retinal world obtained Hyper-spectral imaging is appropriate for screening and diagnosis
by preliminary fundus cameras lacked the ability to apprehend of diseases such as: diabetes, AMD, and glaucoma [20].
depth during examination of fundus image which lead to inaccu- Scanning Laser Ophthalmoscopy (SLO) [21] uses a focused
rate diagnosis of certain retinal pathology (e.g. cotton wool spots). laser beam to illuminate the retinal fundus and provides retinal
Tomography based imaging has resolved this problem. OCT has images with high spatial sensitivity. It is helpful in diagnosis of
now been successfully employed to develop 3-D view of retina retinal disorders such as macular degeneration and glaucoma.
Adaptive Optics SLO [22] utilizes SLO and enhances the results
[15].
by removing aberrations from retinal images through adaptive
optics. The retinal images obtained by adaptive SLO are sharper
2.1.1. Fundus photography
than those obtained by SLO.
The process of obtaining a 2-D representation of 3-D ocular
Angiography is the process of injecting fluorescent dye in
fundus through projection of reflected light on to an imaging
retinal blood vessels and projection of emitted photons from
plane is termed as fundus photography. Fundus photography,
the dye on to an imaging plane. The fluorescent dye, which is
mydriatic or non mydriatic, involves use of a low power intricate
injected in circulation, is sensitive to the light reaching it i.e. the
microscope with an attached camera for imaging the interior
fluorescence is different for different wavelengths of light. This
surface of the eye which includes retina, OD, retinal vasculature,
chameleon behavior of fluorescent dye makes angiography an ap-
posterior pole, and macula. Digital Imaging of inner lining of eye
posite imaging modality for inspection of retinal blood circulation
surface through fundus photography follows the same footings
[23].
as those of conventional image capturing except that instead of
a photographic film, digital cameras have sensors (charge cou- 1. Sodium Fluorescein Angiography (FAG) is a variation of
pled device or complementary metal oxide semiconductor active angiography. However, the fluorescent dye injected in reti-
pixel). Each sensor converts light signal into electric signal and nal blood circulation is sodium. When the retinal blood
result is stored in the form of a single pixel. The resolution of vessels and neighboring retinal tissues are illuminated with
image can be increased by increasing number of sensors. blue light (490 nm wavelength), the dye fluoresces yellow
Advancements in the models of fundus photography have light (530 nm wavelength). It is helpful in documenting
been witnessed since the last century [2]. The modern fundus the retinal manifestaions of cystoid macular edema and DR
photography includes the following imaging modalities: [23].
Color fundus photography modifies spectral range of illu- 2. Indocyanine Green Angiography (ICG) is another variant
mination source through the use of contrast filters (red, blue, of angiography which employs indocyanine as fluores-
and green filters). This variability in spectral range of illuminat- cence agent. The injected indocyanine glows only in in-
ing source enhances visibility of several ocular structures. For frared spectrum thereby making the blood, fluids, and
instance, blue color improves perceptibility of anterior retinal choroidal pigments, present in posterior part of eye, trans-
layers, which appear transparent in the presence of white light. parent for the observer. This behavior makes it appropriate
4 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

Fig. 1. Retinal fundus images (a) OD centered (b) Macula centered.

for diagnostic analysis of deep choroidal disorders such Ophthalmic artery provides the circulation to the retina. Reti-
as choroidal neovascularization, abnormal vessels supply- nal blood vessels exist in the form a network of arterioles and
ing ocular tumors, and hyperpermeable vessels leading to venules which circulate the whole retinal region. Due to nonfen-
central serous chorioretinopathy [24]. estration of the vascular endothelium a blood–retina barrier is
maintained by retinal blood vessels. Microaneurysms and hem-
A diverse set of datasets based on retinal fundus photography, orrhages are produced as a result of damage to this blood–retinal
from different ethnicities, has been developed. Some of them are barrier. Moreover, out pouching of retinal blood vessels appear as
publicly available while others can be obtained on demand. A neovascularization disturbing their morphology. Various systemic
brief overview of these datasets is presented in Table 1. and retinal diseases manifest themselves in the form of variation
in retinal blood vessels’ features.
2.1.2. 3-D optical coherence tomography (OCT)
OCT [25] is a widely accepted noninvasive imaging modal- 2.3. Disease manifestations as retinal impairments
ity among ophthalmologists. This technique captures 3-D cross-
sectional maps of retina by utilizing the principle of interferom- Analysis of retinal morphology provides an insight to general
etry and confocal microscopy. Accuracy of 3-D OCT images is health state of entire human body as many systemic and retinal
usually 10 to 15 µm. Depending on the light source used, res- diseases manifest themselves as ocular structures’ impairments.
olution of images can be improved. The illumination source with Following rubrics provide a compact overview of such diseases.
large wavelength is preferable because it provides efficient scat-
tering of light in retinal cross-section. These images are efficient 2.3.1. Diabetic retinopathy (DR)
in diagnosis of macular pucker, vitreomacular traction, macular Diabetes mellitus (DM) is a disease caused by fluctuating levels
hole. However, OCT has recently been regarded as a new standard of glycemia (hypoglycemia, hyperglycemia). Repercussions of DM
for detection of diabetic macular edema. The advancement in can emanate as asymptomatic development of DR (damage to
technologies has made OCT capable of generating angiograms for the eyes), diabetic nephropathy (damage to kidneys), and diabetic
assessment of retinal vasculature. neuropathy (damage to nerve) [27]. The most prevalent conse-
quence of DM is DR. Despite the use of intensive glycemic control
2.2. Retinal anatomical structures therapy DR remains a vision threatening complication of diabetes.
Severe symptoms of DR include development of microaneurysms,
Human eye, which is mostly a hollow organ, is internally neovascularization, hemorrhages, cotton wool spots, and exu-
lined by light sensitive tissues collectively named as the retina. dates in the retinal region. Extreme progression of polygenic
The posterior pole of the eye is tightly knit with neuronal cells, disease puts the patient’s eyesight at stake [28]. Pathogenesis like
photoreceptor cells, and glial cell. Rods, cones and ganglion cells capillary closure and dysfunction of blood–retina barrier are the
are three types of photoreceptor cells. The optic nerve fibers main anatomical changes in retina that lead to DR. Their brief
originate from nerve fiber layers which emanate from axons of detail is mention in the following sections.
ganglion cells. The ocular structures focus the image on retina 2.3.1.1. Retinal vessel closure. Many retinal disorders stem from
and nerve fibers transfer the information to brain in the form obliteration of retinal capillaries such as neovascularization is
of electrical signals which then interprets the received signals as reported to be preceded by nonperfusion and closure of small
visual images. Optic nerve head or OD, oval in shape, serves as retinal blood vessels [29]. Retinal vessel closure instigates the
the exit point for nerve fibers and is located 4.5 mm to 5 mm development of neovascularization, microaneurysms, cotton wool
nasal to the center of retina as depicted in Fig. 1. The maximum spots, and hemorrhages.
resolving power of the eye comes from the anatomical center of Neovascularization Retinal blood vessel closure leads to limited
eye i.e. macula. Fovea and foveola are resent in the macular region supply of oxygen to retinal regions corresponding to these ves-
and both of them contain a large number of photoreceptors sels. This condition is often termed as retinal ischemia [30]. The
(cones) making them specialized retinal region for high acuity vascular endothelial growth tries to compensate the decreased
vision [26]. Visual capacity of human brain is primarily dependent oxygen supply resulting in the formation of weak small blood
on visual input from macula but progressive damage to macula vessels on the surface of retina thereby leading to vision loss.
results in inculcation of diseases like macular degeneration or in Microaneurysms: Limited oxygen supply results in unusual di-
severe cases can create macular hole bursting the blood vessels lation of retinal capillaries. These bulgings of capillary endothelial
coursing towards macula.
Table 1
Specifications of 2-D fundus Retinal Image Datasets.
Sr. Dataset Source Camera Specs. Field of view (FOV) Image Image size (pixels) Number of Ground Truth Availability
format Images
1 DRIVE [31] DR screening test held in Canon CR5 non-mydriatic 45◦ JPEG 768× 584 40 Blood Vessel demarcation Public
Netherlands 3-CCD camera compressed

M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

2 STARE [32] Shiley Eye Center at the TopCon TRV-50 fundus 35◦ – 605x 700 20 Blood Vessel demarcation Public
University of California, San camera
Diego, Veterans Administration
Medical Center in San Diego.
3 ARIA [33] St. Paul’s Eye Unit, Royal Zeiss FF450+ fundus 50◦ Uncom- 768x 576 212 Trace of blood vessels, the Public
Liverpool University Hospital camera pressed optic
Trust, Liverpool, UK and the TIFF disc and fovea location
Department of Ophthalmology,
Clinical Sciences, University of
Liverpool, Liverpool,
UK
4 CHASE_DB1 [6] Child Heart and Health Study Nidek NM-200-D 30◦ TIFF 1280x 960 28 Blood Vessel demarcation Public
in England (CHASE), a
cardiovascular health survey in
200 primary schools in London,
Birmingham, and Leicester
5 DIARETDB0 [34] Captured under IMAGERET Several digital fundus 50◦ PNG 1500x 1152 130 Each image is marked as Public
project cameras having any red small dots,
hemorrhages, hard exudates,
soft exudates,
neovascularization or not
6 DIARETDB1 [35] Captured under IMAGERET Several digital fundus 50◦ PNG 1500x 1152 89 Demarcation of red small dots, Public
project cameras hemorrhages, hard exudates,
and soft exudates
LaTIM laboratory and 1440 x 960
TopCon TR NW 6 3CCD
7 MESSIDOR-2 [36] the Messidor program 45◦ TIFF 2240 x 1488 1748 On demand
fundus camera
partners 2304 x 1536
2544x 1696
ANR-TECSAN-TELEOPHTA
8 E-Ophtha EX [36] - - JPEG 2048x 130 82 Exudates marked Public
project
1440x 960
E-Ophtha MA [36] ANR-TECSAN-TELEOPHTA - - JPEG 2544x 169 381 Microaneurysms marked Public
project 1440x 960
9 NIH AREDS Project funded by NIH – – 5600 Public
10 DriDB [37] university hospital in Zagreb Zeiss VISUCAM 200 45◦ Uncom- 720x 576 50 location of OD, blood vessels, On Demand
fundus camera pressed hard exudates, soft exudates,
BMP dot and blot hemorrhages, and
neovascularization
11 EYE-PACS1 Captured during regular Canon CR1/DGi/CR2, 45◦ JPEG Varying sizes 9963 Each image marked as having: Public
screening of DR affected Centervue DRS, and No DR, Mild, Moderate, Severe
patients under EYEPACS Topcon NW camera or Proliferative DR
program

5
6 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

linings often appear in the form of small sac like structures named 2.3.2. Age related macular degeneration (AMD)
as microaneurysms. The fragility of capillary walls is considered For people of age above 50 years, AMD is found to be the chief
to be the main reason behind development of microaneurysms cause of irreversible vision loss. Presence of AMD is characterized
[38]. They appear as small red dots in retinal images as depicted by excessive presence of drusen, yellow dots, in the macular
in Fig. 2. region of eye. Small number of hard drusen is not regarded as
Cotton wool spots: Oxygen supply to certain retinal areas may symptoms of AMD because people with age more than 50 years
are likely to develop drusens as a normal anatomy of retina. But
completely close off due to blockage in arterioles. Consequently
anomalously large number of drusen can result in mutilation of
large regions of retina become completely deprived of oxygen and
retinal pigment epithelium [41]. AMD can be classified into the
result in emanation of fluffy white patches identified as cotton following categories:
wool spots or soft exudates [38]. Early AMD is marked by the presence of less than 20 medium
Hemorrhages: The blockade in arterioles may instigate a pres- sized drusen or other abnormal retinal pigments [42].
sure build up within the vessels. Significant amount of pressure Intermediate AMD is characterized by the presence of many
could burst the vessels and result in origination of hemorrhages medium sized drusen, one large druse or geographic atrophy that
[38] as shown in Fig. 2. is away from the center of macula [42].
Advanced non-neovascular AMD (Dry AMD) manifests itself in
2.3.1.2. Mutilation of blood–retina barrier. Retinal blood vessels the form of drusen and geographic atrophy that extends to fovea
are permeable i.e. they allow motion of fluids through their i.e. the center of macula [42].
walls. In DR, the retinal vessels become fenestrated and abnormal Advanced neovascular AMD (Wet AMD) is indicated by presence
leakage of blood cells, proteins, water and other large molecules of exudates, neovascularization, and sequelae of neovasculariza-
starts. tion [42] (see Fig. 3).
Hard Exudates: They appear as a consequence of leakage of fats
and proteins along with water from abnormally permeable walls 2.3.3. Cardiovascular diseases
of retinal vessels. Mostly hard exudates appear on the outer layer Coronary heart disease is directly linked with micro-vascular
of retina individually, in the form of patches, or surrounding mi- circulation. Changes in microvasculature, structure and pathology
of human circulation can be studied through in vivo examination
croaneurysms in the form of a crescent. They appear as yellowish,
of retina. Therefore, retinal image analysis provides a window in
waxy, and glistering structures in retinal images as presented in
the health of human heart. Common retinal vascular signs include
Fig. 2.
formation of microaneurysms, hemorrhages, arterio-venous nick-
2.3.1.3. Macular edema. The central region of retina is macular ing, and focal arteriolar narrowing. These signs reflect vascular
region and is responsible for high acuity vision. Macular edema damage because of hypertension, aging, and other processes.
(ME) is a consequence of retinal thickening near macula and Retinal vascular narrowing is found to be associated with reduced
myocardial perfusion measures on cardiac magnetic resonance
fenestration of retinal blood vessels. The anomalous vasoper-
imaging. Moreover, it has also been found that other retinopathy
meability allows abnormal flow of large molecules like water,
lesions are linked to coronary artery calcification. These kind
blood cells, lipids, and proteins through retinal vascular walls. of anatomical and pathological reasons suggest that changes in
Accumulation of such fluids in the macular region results in retinal microvasculature can be useful for assessing the coronary
the development of pigments such as exudates leading to ME. heart disease risk stratification [43].
Treatment of ME is only recommended when the pigmentations
develop at the center of macula i.e. fovea. This condition is termed 2.3.4. Glaucoma
as clinically significant macular edema (CSME). CSME is actually The second most prevalent cause of permanent vision loss
defined as presence of hard exudates within 500 µm of the in the developed world is glaucoma. This disease directly dam-
macular or retinal thickening within 500 µm of the macular ages the optic nerve and the ganglion cells [44]. Optic nerve of
region [39]. Prolonged and asymptotic DM results in the origi- glaucoma affected eye shows cupping of optic nerve up to an
nation of diabetic macular edema preceded by proliferative and abnormal amount. Obesity, high blood pressure, migraines, and
increased pressure in eye are the factors that instigate glaucoma.
non-proliferative DR [40].
Glaucoma is classified as open angle, close angle, and normal
Severity levels of DR have been graded based on the variants
tension glaucoma. Open angle glaucoma is the most prevalent
of pathology present in retina. A brief detail about grading levels type of glaucoma. It grows gradually and is painless but it has the
of DR, in accordance with the severity, is presented below: potential of making the patient completely blind if left untreated.
Proliferative DR (PDR) is marked by presence of retinal hem- However, closed angle glaucoma has the ability to manifest itself
orrhages. slowly or abruptly. Normal tension glaucoma is associated with
Moderate PDR manifests itself in the form of neovasculariza- blood circulation issues in retinal area and other organ perfusion
tion and retinal hemorrhages. whereas the classic hallmark of intra-occular pressure has no
Severe PDR is indicated by the presence of detachment of association with this kind of glaucoma. Glaucoma is diagnosed by
retina and neovascularization on iris. analysis of intra-ocular pressure, cup to disk ratio, retinal vascular
Insignificant non-proliferative PDR (NPDR) signs are microa- morphology, optic nerve structure, and anterior chamber angle
neurysms, retinal hemorrhages, and hard exudates [45].
Significant NPDR has similar signs as those of insignificant
2.4. Quantitative measures for retinal image analysis
NPDR with vessel closure.
High Risk NPDR depicts itself as retinal hemorrhages and An algorithm is considered efficient if its predictions and
vessel closure. ground truth stand in close proximity to each other. This proxim-
Moderate ME is indicated by hard exudates, origination of ity can only be tested by the use of some quantitative measures.
Edema outside fovea region, and thickening of retina. These quantitative measures also help in evaluating and com-
Severe ME manifestations are similar as those of moderate ME paring the capabilities of different algorithms. Sophisticated per-
except that Edema develops inside the fovea region. formance metrics are derived from basic performance measures
 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203 7

Fig. 2. (a) Retinal image with pathologies (b) Hemorrhages (c) Soft Exudates (d) Hard Exudates.

Fig. 3. Retinal image diagnosed (a) with Dry AMD (b) with Wet AMD.

(true positives, false positives, true negatives, and false nega- suggested that instead of hard-coded knowledge, AI systems re-
tives). The performance metrics which are used for analysis and quired to develop self-learning ability from raw data. This concept
comparison of algorithms in this study are recorded in Table 2. surfaced as the field of machine learning. Performance of machine
Ratio of classified exudate pixels to the total pixels plus ratio learning algorithms was primarily dependent on representation
of pixels classified as non-exudates to the total pixels. of data provided to them. Maiden machine learning algorithms
required manual representation of data in the form of features.
3. Deep-learning The algorithms deduced results on the basis of correlation be-
tween features and output. Logistic regression and Naïve Bayes
Creation of self-taught and self-thinking machines has re- are examples of machine learning algorithms.
mained utter desire of humans since antiquity. With the inven- Representation Learning: Hand crafting of features for tedious
tion of programmable computers, scientists remained focused tasks is laborious and time consuming. It may take decades for
towards tackling problems which were intellectually difficult for a community of researchers to devise maximally correlated fea-
humans in early days of artificial intelligence. Artificial intelli- tures for such complex tasks. This problem was tackled by the
gence has coursed through the following phases before turning development of representation learning which is learning of map-
the milestone of self-taught machines [46]: ping from initial representation of data to an intermediate repre-
Knowledge based AI: This approach of AI required the program- sentation and then back to the original representation. A classic
mers to feed the machine with all the details about the problem. example of representation learning is auto-encoder.
Surprisingly the use of knowledge base approach made a num- Deep Learning: Many factors of variation are associated with
ber of monotonous tasks easier for machines to solve. Through learned features in case of representational learning. An example
knowledge base approach to artificial intelligence (AI), computers of factors of variation is the different viewing angles with which
have defeated even the best chess player, Garry Kasparov [47]. an image of a car is captured. Concepts of deep learning or expe-
Machine Learning: Solving intuitive and subjective problems, such rience based learning come handy in solving this crucial problem
as recognizing voices and contents of an image, proved to be of representation learning. Hierarchical nature of deep learning
a taxing task for AI. Solution of such intuitive tasks required techniques allows building of complex concepts out of simpler
immense amount of knowledge about the surrounding world ones. Quintessential example of deep learning technique is feed
and a large database of decision statements. These challenges forward network termed as multi-layer perceptron (MLP). These
8 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

Table 2
Performance metrics for retinal image analysis.
Performance metric Description
Sensitivity (SN) Ratio of classified true positives to the actual true positives in
the ground truth. Also known as true positive rate (TPR)
Specificity (SP) Ratio of classified true negatives to the actual true negatives
in the ground truth. (1-SP) is also known as false positive rate
(FPR)
Receiver Operating Mapping of TPR to FPR at different threshold levels or
Characteristic curve (ROC) mapping SN to SP in a 2 dimensional plane
Area Under the Curve (AUC) Area covered by ROC curve when optimality is achieved

techniques are well established in audio recognition, natural This reconstruction is probabilistic. A number of network vari-
language processing, and automatic speech recognition [48]. ants exist which include: auto-encoders, stacked auto-encoders,
Benefits of deep learning are achieved through the use of stacked de-noised auto-encoders, restricted Boltzmann machines
DNN. It is a form of artificial neural network (ANN) in which ar- etc. Usually noise is added in input images and stacked de-noising
rangement of neurons is inspired by neuron disposition of animal auto-encoder is then used to reconstruct original image from
visual cortex. DNN provide the following major commendatory compressed noisy image.
advantages:
3.3. Challenges to deep learning
1. Hierarchical feature extraction i.e. no need of handcrafted
features. In recent years deep learning techniques have emerged as rev-
2. Limited pre-processing of input images olutionary methods which have surpassed most of the state-of-
DNN have three kinds of layers: input layer, hidden layers, and the-art-methods. The ability of deep networks to exploit simple
output layer. Each layer has non-linear units known as nodes as well as complex compositional features of data representations
which help in modeling of complex features. DNN can learn is referred to as the reason behind their success. Notwithstanding,
features by following either the supervised learning method or there are some issues with deep learning techniques which are
the unsupervised method. A brief introduction of these methods yet to be solved.
is listed under the following rubrics.
3.3.1. Lack of real cognizance behind deep learning techniques
3.1. Supervised learning Despite achieving remarkable performance, the real
cognizance behind the achievements of deep learning has re-
In supervised method DNN is provided with training data mained unknown. A small number of efforts have been put forth
along with output labels and network tries to learn labeling using by researchers to understand how deep networks achieve such
a specified learning method under the supervision of available exceptionally good results. However, there is a utmost need to
ground truths. Classification problems are solved using super- develop a comprehensive theoretical background about tuning
vised learning approach. and performance assessment strategies of deep networks [49].
As cortical neurons in animals consider a restricted region of The following questions are still unanswered.
space to generate response, similarly neurons of CNNs respond i. How to choose features to be extracted?
to a restricted region of input image known as receptive field. ii. How to tune parameters of network models?
This receptive field can be viewed as an image processing filter
or kernel. Receptive fields are overlapping, to give an effect of 3.3.2. Difficult training
sliding over input image. Response is generated by convolution Training of DNN has always remained a difficult task as there
of receptive field of neuron with weight matrix (generated by are numerous chances that they network may get stuck in local
individual weights of neurons in same layer). Each node of input optima. Moreover, training of deep networks is very slow and
layer gets an input image patch, and output is generated for requires massive computational resources for both medium and
central pixel of patch as shown in Fig. 4. Output label maps for all large sized data-sets to reach performance of state-of-the-art
pixels of input image patch can also be generated simultaneously performance especially in case of offline learning. Once trained
using structured prediction method. the network becomes non-adaptive for new data [49].
Usually CNNs have convolutional layers, pooling layers, and a
terminating classification layer. Pooling layers are used to reduce 4. Learning methods applied for retinal image analysis
dimension of feature vector which further increases computa-
tional efficiency of the network as shown in Fig. 4c. Softmax, In this survey, we have reviewed 34 articles which have
Linear Support Vector Machine (LSVM), multi-class SVM, random demonstrated the use of deep learning based techniques for
forest etc. can serve the purpose of classification layer. CNNs have classification/segmentation of retinal anatomical structures and
many variations depending on number of layers, pooling, and detection of retinal diseases. A gist of every article is presented
classifier. These include LeNet-5, VGG-16, and GoogleNet etc. which encompasses the main focus of the reviewed manuscript.
The proposed deep learning based algorithms are classified into
3.2. Unsupervised learning two main classes i.e. (i) supervised learning algorithms (ii) un-
supervised learning algorithms. However, these two main classes
This approach is used for pattern recognition. Unsupervised are further categorized into the following types: (i) Retinal vas-
DNNs also have input layer, hidden layers, and output layer. culature classification/segmentation (ii) Exudates Classification/
These layers can be partially or fully connected, we can also have Segmentation (iii) Microaneurysm Classification (iv) Simultane-
a terminating fully connected classification layer. Unsupervised ous segmentation of retinal anatomical structures (v) Simulta-
DNN take input image and compress it, their principle is to neous segmentation of retinal pathologies (vi) DR Identification/
reconstruct input image from compressed version of input image. Classification (vii) Retinopathy of Prematurity Identification (viii)
 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203 9

Fig. 4. Deep Neural Networks (a) Receptive field of CNN (b) Supervised CNN (c) Structure of CNN (http://swinghu.github.io/deep_learning/2015/10/09/dl-tutorials.html).

AMD Identification/Classification (ix) Multiple Retinal Disease layers and last fully connected layer are fed into three random
Classification. Table 3 shows a summary of the reviewed articles forest classifiers. After training of CNN, each random forest is
including the author names, year of publication, data sources trained with learned features extracted from the network. An
used for validating the proposed algorithms, proposed method- ensemble classifier with winner-takes-all strategy has been em-
ology, and the main focus. The methodology followed by each ployed after complete training of random forest classifiers. DRIVE
manuscript is discussed in the following subsections. Moreover, and STARE datasets are used to test efficacy of proposed method.
a quantitative performance comparison table is provided at the Table 4 shows average case performance metrics of algorithm for
end of each subsection which provides the detail of the database both datasets. Robustness of algorithm is inspected using cross
sources used for performance evaluation and statistical results training. Accuracy in case of DRIVE (trained on STARE) is 98.03%
in terms of sensitivity, specificity, accuracy, and area under ROC and for STARE (trained on DRIVE) accuracy is 97.10%.
curve. Classification of each pixel as vessel or non-vessel poses two
problems: (i) system is not robust to pathology presence and (ii)
4.1. Segmentation of retinal vasculature pixel–pixel training of network is exhaustive. These problems can
be efficiently handled by structured prediction i.e. output label
4.1.1. Supervised methods map for all pixels in input image patch instead of a single label
Supervised Retinal vessel segmentation problem can be bro- for central pixel. This approach is presented in detail by Fang
ken down into two stages: (i) feature extraction, (ii) classification. et al. [51]. Their DNN has five layers with three hidden layers,
Wang et al. [50] have exploited the same approach with CNN as a network is made wide to make feature extraction effective. To
hierarchical feature extractor. They used ensemble random forest, allow learning of complex relationship between cross-modalities,
which has good generalization ability, as a classifier. This classifier input image patch and output label map, network is made deep.
gives a binary output for each pixel as vessel pixel or background The label map is used to construct probability map which is of
pixel. The green channel of RGB images contains the best retinal the same size as that of original retinal image. Probability map is
vessel and background contrast therefore green channel of images enhanced by histogram equalization. Enhanced probability map is
is used in the proposed methodology. Histogram equalization and smoothed with Gaussian kernels of varying standard deviation to
Gaussian smoothing filter are utilized to reduce the noise and get different scale space images. This approach is more objective
normalize uneven illumination in images. Training of network is and results are not affected by image noise and other unwanted
compute intensive therefore super-pixel based sample selection image artifacts. Results are tested on DRIVE dataset and compared
is done using Simple Linear iterative clustering (LISC). The net- with state-of-the-art SIFT and COSFIRE methods. The proposed
work has five layers (classic LeNet5) with one fully connected method has superseded the traditional methods in identification
output layer. Features learned from intermediate subsampling of bifurcation points, extreme curvature and cross-over points.
10 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

Table 3
Deep learning based retinal image analysis methods.
Algorithm Year Dataset Method Focus
Wang et al. [50] 2015 DRIVE, STARE
Fang et al. [51] 2015 DRIVE
Melinščak et al. [52] 2015 DRIVE
Fu et al. [53] 2016 DRIVE, STARE,
CHASE_DB1
Liskowski and 2016 DRIVE, STARE
Krawiec [54]
Supervised
Li et al. [55] 2016 DRIVE, STARE,
Retinal vasculature
CHASE_DB1
Classification/
Yao et al. [56] 2016 DRIVE Segmentation
Dasgupta and Singh 2016 DRIVE
[57]
Welikala et al. [58] 2017 UK Biobank
Xu et al. [59] 2018 DRIVE
Maji et al. [60] 2015 DRIVE Healthy and Abnormal
Li et al. [55] 2016 DRIVE, STARE, Hybrid Retinal Anatomy
CHASE_DB1 Classification/Segmentation
Lahiri et al. [61] 2016 DRIVE Unsupervised
Prentašić and 2016 DRiDB
Lončarić [62] Exudates Classification/
Supervised
Abbasi-Sureshjani 2017 DR2, DIARETDB1, Segmentation
et al. [63] E-Ophtha-EX
Gulshan et al. [64] 2016 DIARETDB Unsupervised Microaneurysm Classification
Maninis et al. [65] 2016 DRIVE, STARE,
DRIONS-DB,
RIM-ONE
Simultaneous
Tan et al. [66] 2017 DRIVE Supervised segmentation of retinal
Zilly et al. [67] 2017 DRISHTI-GS, Messidor, anatomical structures
RIM-ONE
Tan et al. [68] 2017 CLEOPATRA
Lam et al. [69] 2018 E-Ophtha Simultaneous
Supervised segmentation of retinal
Badar et al. [70] 2018 Messidor
pathologies
Pratt et al. [71] 2016 Kaggle
Abràmoff et al. [72]) 2016 MESSIDOR-2
Gulshan et al. [64] 2016 EyePACS-1,
MESSIDOR-2
DR Identification/
Colas et al. [73] 2016 Kaggle DR dataset Supervised
Classification
Gargeya and Leng 2017 MESSIDOR,
[74] E-OPHTHA
Retinal Disease
Worrall et al. [75] 2016 Private dataset Supervised Retinopathy of Prematurity
Classification
Identification
Burlina et al. [76] 2016 NIH AREDS
Burlina et al. [77] 2017 NIH AREDS AMD
Supervised
Identification/Classification
Lee et al. [78] 2017 Private Dataset
Choi et al. [79] 2017 STARE Supervised
Multiple Retinal Disease
Arunkumar and 2017 ARIA Unsupervised Classification
Karthigaikumar [3]

Melinščak et al. [52] also presented a DNN for classification maps by a fully connected four layered CNN named as Deep-
of retinal vasculature. Their Network has four blocks of alternat- Vessel. Convolutional layers of the proposed network have ReLU
ing convolutional and max pooling layers with two fully con- activation function. The network is a modification of holistically-
nected layers at the end. Convolutional layers have ReLU activa- nested edge detection (HED) network [80]. Probability maps from
tion function and last fully connected layer has Softmax activation all side-output layers are fused together and a single probability
function. The DRIVE dataset is used for training of network. map is generated. The fully connected Conditional Random Fields
Vessels are extracted from green channel of raw input images (CRF) are used for precise localization of vasculature boundaries.
i.e. no pre-processing. Moreover, Backpropagation algorithm is CRF is formulated as a Recurrent Neural Network (RNN). Max-
used for training of network. The statistical measures obtained imum posterior marginal inference is obtained through mean
after testing of network are shown in Table 4. field approximation of CRF distribution. The network is fine-tuned
Fu et al. [53] formulated the problem of vessel extraction with ARIA dataset. Results obtained after evaluation of algorithm
as a boundary detection problem. They constructed probability using DRIVE and STARE datasets are presented in Table 4.
 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203 11

Table 4
Performance measures for classification/Segmentation of Retinal vasculature using Supervised/Hybrid/Unsupervised learning Methods.
Dataset SN SP Acc AUC Method
Wang et al. [50] DRIVE 81.73% 97.33% 97.67% 94.75%
STARE 81.04% 97.91% 98.13% 97.51%
Melinščak et al. [52] DRIVE – – 94.66% 97.49%
DRIVE 76.03% – 95.23% –
Fu et al. [53]
STARE 71.42% – 95.85% –
CHASE_DB1 71.30% – 94.89% –
DRIVE 75.69% 98.16% 95.27% 97.38% Supervised
Li et al. [55]
STARE 77.26% 98.44% 96.28% 98.79%
CHASE_DB1 75.07% 97.93% 95.81% 97.61%
Yao et al. [56] DRIVE 77.31% 96.03% 93.60% –
Dasgupta and Singh [57] DRIVE 76.91% 98.01% 95.33% 97.44%
Liskowski and Krawiec DRIVE 81.49% 97.49% 95.30% 97.88%
[54] STARE 90.75% 97.71% 97% 99.28%
Welikala et al. [58] UK Biobank – – 86.97% –
Xu et al. [59] DRIVE 87% 98% – –
DRIVE 75.69% 98.16% 95.27% 97.38%
Li et al. [55]
STARE 77.26% 98.44% 96.28% 98.79%
Hybrid
CHASE_DB1 75.07% 97.93% 95.81% 97.61%
Maji et al. [60] DRIVE – – 92.37% –
Lahiri et al. [61] DRIVE – – 95.3% – Unsupervised

Another supervised learning method for segmentation of reti- Neighboring pixels, in normalized image patches, are uncorre-
nal vasculature has been proposed by Yao et al. [56]. Their lated by multiplication of data matrix with whitening matrix.
network contains three convolutional layers, three intermediate To increase the number of input samples, normalized and un-
pooling layers, one fully connected layer and a terminating soft- correlated input patches are augmented. Six different types of
max classification layer. The preliminary weight initialization is CNN architectures are developed: PLAIN, GCN, ZCA, AUGMENT,
done with Gaussian distribution of specified mean and variance NO-POOL, and BALANCED. As the name suggests NO-POOL net-
whereas further weight learning is done via backpropagation. The work has no intermediate pooling layers, it has four preliminary
output of softmax layer is confidence level, being blood vessel convolutional layers and four fully connected layers. The PLAIN
pixel, with values in the range (0, 1). Classification is perfected by architecture has two initial convolutional layers followed by a
two stage binarization based on three dimensional characteristics max-pooling layer which is further followed by 2 convolutional
of input image; confidence level image from CNN, green channel layers, one max-pooling layer and three fully connected layers.
of RGB retinal input image, and their difference. The initial bina- GCN, ZCA, AUGMENT architectures are exact replica of PLAIN,
rization is done by global thresholding of image which classifies they differ in training set-up. For BALANCED network, PLAIN
each pixel as belonging to vessel class or non-vessel class. The network is trained with equal share of decision classes. Output
local binarization is employed with respect to pixel-class centroid units have sigmoidal activation whereas hidden layers have ReLU
distance. This two stage binarization results in formation of activation units. Single pixel based segmentation method is ver-
unwanted artifacts which can be removed through morphological ified on DRIVE AND STARE datasets and results are presented in
post processing i.e. erosion. Structuring elements of varying sizes Table 4.
are used for post processing procedure. Efficacy of algorithm is For structured prediction, network architecture is same as that
validated using DRIVE dataset. Results, as shown in Table 4, are of NO-POOL architecture except that the last layer is replaced
comparable to human observer performance measures. by s2 neurons and is fully connected with the second last layer.
Dasgupta and Singh [57] provided an improved version of This set-up is done for construction of label map. BALANCED and
supervised structured prediction based multi label method for NO-POOL configurations are considered for structural prediction.
retinal vessel segmentation. Their CNN has 6 layers including a DRIVE and STARE datasets are used for validation of algorithm;
max pooling layer, an unsampling layer, and a final fully con- results are shown in Table 4.
nected softmax layer. Each convolutional layer uses ReLU as an Structured prediction is more sensitive to false negatives i.e. it
activation function. Input layer of network is fed with prepro- is less sensitive to segmentation of fine vessels. On the contrary
cessed green channel extracted image patches. Pre-processing non structured prediction is reliable for reconstruction of retinal
steps include: image normalization, Contrast Limited Adaptive vasculature because it gives less false negative errors.
Histogram Equalization (CLAHE), and gamma adjustment. The Classification of retinal vessels into arterioles and venules also
DRIVE dataset is used for training and testing purposes. This plays a pivotal role in diagnosis of many systemic, microvascular,
dataset has small number of images therefore; neuron dropout and ophthalmic diseases. Welikala et al. [58] proposed a robust
method is used for increasing robustness of network. The results deep learning based method for classification of arterioles and
obtained after testing are depicted in Table 4. venules across the entire retinal image considering the vessels
Liskowski and Krawiec [54] have compared single pixel clas- located at OD. Proposed network contained 6 layers including 3
sification and structured prediction (SP) based segmentation of convolutional layers and 3 fully connected layers. This method
vessels. This extensive study is based on different network archi- was evaluated on the UK Biobank (large population-based co-
tectures and pre-processing techniques. The uneven illumination hort study of middle aged and older adults) dataset. In case of
in input image patches is normalized through global contrast arterioles classification, the network achieved 86.07% of sensi-
normalization (GCN). Zero phase component analysis (ZCA) is tivity, 87.67% of specificity, and 86.97% of accuracy whereas, for
employed for the purpose of single pixel based classification. venules classification it achieved 87.67% of sensitivity, 86.07% of
12 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

specificity, and 86.97% of accuracy. Results are also presented in diversification is added by parallel training of two E1.nets with
Table 4. different architecture. Convex weighted average is used to get the
Xu et al. [59] proposed a fully convolutional neural network combined decision of both E1.nets. The algorithm is tested using
inspired from U-net architecture for simultaneous segmentation publicly available DRIVE dataset. The maximum average accuracy
of arterioles and venules from retinal image. Proposed network is recorded at 95.3% and Kappa agreement coefficient at 70.9%
takes colored image as input and outputs arterioles and venules as presented in Table 4. This method outperformed the human
with colored labels. The network was trained and tested on observer accuracy which was recorded at 94.7%.
publicly available dataset DRIVE and outperformed the other
state-of-the-art methods. Sensitivity and specificity of 87% and 4.2. Segmentation of exudates
98% respectively are achieved for overall vessel segmentation as
illustrated in Table 4. The misclassification rates of 9.8% and 23.7% 4.2.1. Supervised methods
are recorded for venules and arterioles respectively. Exudates, lipids and lipoproteins are pre-eminent signs of
DR. The exudates are of varying sizes, they can vary from very
4.1.2. Hybrid methods small size to the size as large as that of an OD. DNNs provide
The literature provides combination of supervised and unsu- worthwhile provisions for detection of exudates in retinal fun-
pervised methods for segmentation of retinal blood vessels. Such dus images. Prentašić and Lončarić [62] have presented a DNN
an approach has been proposed by Li et al. [55]. Their DNN has architecture in their study for automatic detection of exudates.
five layers with three hidden layers; initially weights of first Their network has ten alternating convolutional and max pooling
layer are obtained by pre-training of a de-noising auto-encoder layers. A Softmax activation function is used after the last layer
(DAE) while the rest of the weights are randomly initialized. to provide probability based classification of individual pixels
This unsupervised technique is used for weight initialization be- i.e. exudate or non-exudate class. Network is fed with green
cause naïve learning of weights through backpropagation has plane of RGB retinal image. They combined probability maps
high probability of erroneous convergence into local minima. from DNN with other landmark detection probability maps to
After weight initialization, overall learning and fine tuning of exploit the domain knowledge i.e. exudates do not appear inside
weights is done using backpropagation algorithm. Network is retinal blood vessels and OD. The OD detection probability map
trained with a batch size of 100 and 30 epochs. Instead of using is obtained from an ensemble of strategies which include: en-
hessian matrix, vessel segmentation is done via thresholding of tropy based approach, Laplacian of Gaussian approach, brightness
probability map. The proposed methodology is tested through approach [81], and Hough transformation method. Blood vessel
ROC analysis using three datasets: DRIVE, STARE, and CHASE_DB1 probability map is generated by using Frangi [82] vesselness filter.
and results are presented in Table 4. However, methodology is After blood vessel and OD detection a parabola is fitted at the
also tested by cross training of network i.e. testing of one dataset center of OD which encapsulates the regions where there is less
on network trained by other two datasets. With this experiment, potential of exudate presence. At the time of training, input image
accuracy measure decreased from 95.81%, 96.28%, and 95.27% is de-noised using total variation (TV) regularization de-noising.
to 94.17%, 95.35%, and 94.85% for the CHASE_DB1, STARE and DRiDB data is used for training and testing of deep network.
DRIVE datasets when they are trained on DRIVE, CHASE_DB1 and Table 5 shows the statistical results obtained after testing.
STARE datasets, respectively. Cross training gives an estimate for An arrangement of CNN, which is a modification of bag-of-
feasibility criteria of algorithm from clinical point of view. visual-words (BoVW) method, for detection of DR is communi-
Another hybrid methodology is formulated by Maji et al. [60] cated by Costa and Campilho [83]. The significance of the method
using an unsupervised training followed by supervised classifier lies in the advantage that the method is capable of performing
training for retinal vessel segmentation. They have used two DAEs feature extraction, feature encoding and classification jointly un-
followed by an RF classifier for extraction of vessels. The unsuper- like the traditional BoVW. They used DR2 and Messidor datasets
vised pre-training of stacked auto-encoder is used for learning to check the usefulness of their method.
weights of network. Once pre-training is done, an ensemble RF Deep learning in small data regime i.e. training of time con-
classifier is trained in a supervised manner in the presence of suming and high sample complexity algorithms using limited
ground truth. The presented algorithm is tested on DRIVE dataset. resources is the need of the hour in medical image analysis. This
The maximum accuracy obtained is 92.37% as depicted in Ta- kind of methodology has been proposed by Otálora et al. [84]
ble 4. The results obtained are not good compared to traditional in their work for exudate classification. They provided a label
methods but they give a state-of-the-art for hybrid DNNs. efficient CNN along with an active learning algorithm, expected
gradient length (EGL). EGL provided a degree of freedom to the
4.1.3. Unsupervised methods learner by allowing selection of instances to be labeled and added
Retinal vessel segmentation can also be done using unsuper- to the training set from a set of unlabeled instances. EGL per-
vised deep learning methods, although this field is not that much formed better than the usual stochastic gradient descent because
explored but it has the potential to surpass statistical measures it helped in earlier convergence of CNN training by feeding the
obtained from current state-of-the-art methods. network with the most informative patches from retinal images.
Lahiri et al. [61] has presented a two level SDAE network The CNN model was based on LeNet network and it was trained
for retinal vessel segmentation. CLAHE is employed to get rid using transfer learning approach. Moreover, the efficacy of the
of uneven illumination. A simple DAE has three layers with one algorithm was tested using a publicly available data-set E-Ophtha.
hidden layer. Random noise is deliberately added in the input to However, this algorithm poses a computational drawback when
achieve unsupervised learning. L-BFG (Limited Memory Broyden– the number of unlabeled instances are large.
Fletcher–Goldfarb–Shanno) is used to update the weight matrix. The number of false positives in the segmentation of exu-
SDAE is obtained by taking hidden layer output of first DAE dates from retinal images can be reduced by using importance
as input to second DAE and training it. The first level of en- sampling approach. This approach has been followed by [63]
semble is formed by making a collection of n DAEs termed as in their work for boosted exudates segmentation. Importance
E1.net. Multiple DAE kernels allow the vessel feature learning sampling is concerned with prioritizing the sampling towards
in multiple directions. An SDAE (2 hidden layers) with Softmax non-exudate samples that could mislead the network during
classified output makes the second level of ensemble. Further training. This avoids the need of any post-processing steps. The
 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203 13

Table 5 scrutinized using DRIVE and STARE datasets whereas OD segmen-

Performance measures for Classification/Segmentation of Exudates using tation is tested using DRIONS-DB and RIM-ONE dataset. For DRIVE
Supervised learning Methods.
dataset the method achieved 82.2% F1-measure and for stare F1-
Dataset SN SP Acc AUC
measure is recorded at 83.1%. OD segmentation using DRIONS-DB
Prentašić and Lončarić DRiDB 78% – – –
dataset achieved 97.1% F1-measure while that of RIM-ONE is
[62]
found to be 95.9%.
DR2 – – – 97.2%
Abbasi-Sureshjani Tan et al. [66] have recommended and tested a CNN based
DIARETDB1 – – – 96.5%
et al. [63] deep network for concurrent segmentation of OD, fovea and
E-Ophta-EX – – – 99.4%
retinal blood vessels. Three convolution layers with intermediate
max pooling layers constitute the model. Each layer has leaky
Table 6
ReLU activation units. Last fully connected layer has Softmax
Performance measures for Classification/Segmentation of Microaneurysms using
Unsupervised learning Methods. activation unit. Layer one weights are initialized using Xavier
Dataset SN SP Acc AUC
initialization algorithm [86]. Network is trained using backprop-
agation and stochastic gradient descent. Before any classification
Gulshan et al. [64] DIARETDB1 – 91.60% 91.38% 96.20%
input images are normalized using L channel of LUC color space,
then images are converted back to RGB color space and green
channel is extracted for further processing. Training and testing
proposed methodology was based on a CNN with 9 ResNet blocks. of network is conducted on DRIVE dataset and Outcomes are
The retinal images were enhanced using Gaussian kernel before articulated in Table 7.
feeding them to the network whereas the network weights were The segmentation of OD and cup is very important for finding
optimized using stochastic gradient descent [85]. Table 5 contains the Cup to Disc Ratio (CDR) which is an essential indicator of
the statistical results obtained by evaluating the algorithm. The glaucoma progression. Zilly et al. [67] have presented a CNN
testing of the network resulted in AUC of 97.2%, 96.5%, and 99.4% architecture based on ensemble learning. Instead of backpropaga-
when it was evaluated against three publicly available data-sets tion, they have used a greedy approach in which boosting is used
i.e. DIARETDB1, DR2, and E-Ophtha-EX respectively. to learn weights of each stage. Moreover, for training the network,
they have introduced the use of entropy sampling as a substi-
4.3. Segmentation of microaneurysms tute of uniform sampling. Entropy sampling helps the network
in focusing on the most informative points in the image. Also,
4.3.1. Unsupervised methods because of entropy sampling their network has the capability to
Microaneurysms rupture retinal blood vessels and make blood be trained using a smaller dataset. They have investigated the
to leak from them. These miniature lesions appear as early signs efficacy of their algorithm using DRISHTI-GS [87], Messidor and
of DR, therefore, they play a pivotal role in timely detection RIM-ONE datasets. For RIM-ONE dataset, the proposed methodol-
of DR. Automatic extraction of these lesions from 2-D fundus ogy recorded sensitivity, specificity and accuracy of 92.3%, 95.6%,
images has been demonstrated by Gulshan et al. [64]. Extrac- and 94.1% for glaucoma classification using the CDR values.
tion of microaneurysms features is done straightaway from raw Simultaneous segmentation of retinal pathologies can also be
done using multi class CNNs. This approach has been adopted
image patches i.e. no pre-processing is applied. Image patches
by Tan et al. [68] in their work. They formulated a 10 layered
are segregated into two classes i.e. lesion present and non-lesion
neural network for automated segmentation of exudates, mi-
present. Sampling of image patches is done from green channel
croaneurysms, and hemorrhages from retinal fundus images. The
of input image. Computer-aided feature extraction is done with
leaky rectified linear unit was employed as an activation function
the help of SSAE and Softmax classifier is used to label features.
for convolutional and fully connected layers whereas softmax
Two hidden layers constitute SSAE. First layer is used to extract
function was used for final fully connected layer. Moreover, the
features from raw image patches and second layer learns intricate
network was trained using back propagation and stochastic gradi-
features from output of first layer. Softmax binary classifier is
ent descent and regularization was espoused to avoid overfitting.
connected with second layer of SSAE. Algorithm is scrutinized
The retinal images were normalized for uneven illumination cor-
with the help of DIARETDB1 dataset. Classification results from
rection and local contrast adjustment before feeding them to
Softmax classifier are tuned by varying patch-sizes. Fine tuning
network. This CNN based technique is data-dependent because
has provided steady state and more efficient results as compared
the deep learning techniques crave for a large number of training
to previous ones. Results, as depicted in Table 6.
samples. The proposed algorithm is evaluated with a new dataset
named as CLEOPATRA and the results are illustrated in Table 7.
4.4. Simultaneous segmentation of retinal landmarks and patholo- Lam et al. [69] presented a method of localizing and delineat-
gies ing multiple types of pathologies in retinal images. They trained
standard CNNs like VGG-16, AlexNet, ResNet, GoogLeNet, and
4.4.1. Supervised methods Inception-v3 on selected image patches from Kaggle dataset. The
Simultaneous segmentation of retinal landmarks as well as final CNN is selected in the basis of the obtained accuracy. The
pathologies is also possible through the use of DNNs. Maninis method is used to predict the presence of exudates, cotton wool
et al. [65] presented the problem of landmark detection as image– spots, hemorrhages, microaneurysms, and neovascularization in
image regression problem. They formulated a variation of VGG-16 the test image patches. The methodology is validated using the
DNN for segmentation of OD and retinal blood vessels. The net- E-Ophtha dataset. In case of pixel wise classification, the pro-
work is inspired from VGG except that last fully connected layer is posed algorithm attained AUC of 94% and 95% whereas with
removed. It has 5 stages with multiple convolutional layers with lesion-wise classification AUC of 86% and 64% is achieved for
ReLU activation and intermediate max pooling layers. Network microaneurysms and hard exudates, respectively. The obtained
weights are learned by backpropagation algorithm and stochastic performance measures are also presented in Table 7.
gradient descent. To increase generalization ability of network; A more efficient method for simultaneous segmentation of
dataset is augmented with rotated and scaled versions of input retinal pathologies has been presented by Badar et al. [70]. They
images. Efficiency of algorithm towards blood vessel detection is proposed an encoder decoder based fully deep CNN inspired by
14 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

semantic segmentation network Segnet [88]. Proposed network normalization. Algorithm is inspected using two operating points.
was used to semantically segment retinal pathologies i.e. exu- Focus of first operating point is high specificity and that of second
dates, hemorrhages, and cotton-wool spots and assign each pixel is high sensitivity. Results obtained are shown in Table 8.
of the retinal image a class label such as background, exudates, Clinician could be aided in grading of DR through a visualiza-
hemorrhages, and soft exudates. The network was trained and tion heatmap. Gargeya and Leng [74] have provided an automated
tested on a subset of publicly available dataset Messidor and CNN based model for grading of DR and they also generated
achieved state-of-the-art accuracies. The segmentation results heatmap for easy abnormality detection by clinicians. Network
obtained by the proposed methodologies are recorded in Table 7. has convolutional blocks with 4, 6, 8, and 10 layers. Each layer
incorporates batch normalization and ReLU activation units. Last
4.5. Retinal disease classification layer is Softmax classification layer preceded by an average pool-
ing layer and a visualization layer. Heatmap is generated us-
Distinctive characterization of retinal diseases requires expe- ing visualization layer which has functionality similar to that
rienced clinicians but the advancement in the fields of image of a convolutional layer. Training of model is done on EyePACS
processing and machine learning helped in completely automat- dataset. Input images are normalized, downsized and augmented
ing this process. As the traditional methods for automatic clas- to reduce the unnecessary varying brightness and contrast. Some
sification of retinal diseases reached excellence, advent of deep meta-data, related to original image, is appended with feature
learning has provided afresh way for efficient and accurate diag- vector to normalize the effect of environmental variables. Al-
nosis of retinal diseases. The literature has provided supervised gorithm provided 97% AUC, with 94% and 98% sensitivity and
and unsupervised methods for automatic classification of retinal specificity respectively for EyePACS dataset. Generalization ability
diseases using deep learning techniques. The details of the so far of algorithm is tested using MESSIDOR-2 and E-OPTHA datasets.
proposed works are discussed in the proceeding sections. Results are presented in Table 8.
Retinopathy of Prematurity (ROP) is one of the largest pre-
4.5.1. Supervised methods ventable cause of blindness in children. Therefore, early diagnosis
An automated screening algorithm, using supervised deep of ROP plays an imperative role in the prevention of childhood
learning technique, for classification of DR into severe, prolif- visual loss. Worrall et al. [75] have recommended a deep CNN for
erative, and non-proliferative is recommended by [73]. They detection of ROP. They are the first who have proposed a deep
developed a DNN for this purpose. The algorithm is tested using learning based end to end system for early diagnosis of ROP. They
Kaggle DR Detection challenge dataset. This dataset has 10,000 fine-tuned the pre-trained GoggleNet and used it as a detector
images captured from 5000 patients. The algorithm achieved of ROP. The accuracy of the CNN is improved with the help of
an AUC of 94.6%, sensitivity of 96.2%, and 66.6% specificity as a Bayesian framework. They have also employed pre-processing
presented in Table 8. and augmentation of retinal images. The network is trained and
Pratt et al. [71] have provided a DNN for identification and tested using a private database. The statistical performance evalu-
classification of DR into four categories: mild DR, moderate DR, ation resulted in 93.6%, 95.4%, and 94.7% of sensitivity, specificity,
severe DR, and proliferative DR. Their deep network extracts deep and accuracy respectively as shown in Table 8.
features from retinal image which are then used for classification Identification of AMD is a crucial task. Silent nature of the
of DR. The proposed network has ten convolutional layers with disease, during intermediate stage, results in asymptotic severity
intermediate max pooling layers followed by two fully connected leading to complete vision loss. Features learnt from pre-trained
and a Softmax classification layer. Each layer has leaky ReLU neural networks can help in effective diagnosis of AMD at inter-
activation unit and L2 normalization for weights and biases. Color mediate stage. Burlina et al. [76] have checked appropriateness of
normalization, using OpenCV package, is done to even out unnec- this technique for classification of AMD. Overfeat Features (OF)
essary variations in input images. Gaussian initialized weights are are extracted from pre-trained deep CNN on a generic dataset,
batch normalized after each layer. To avoid over fitting, weighted ImageNet. Resized images are fed into the network for learning
class weights and node dropout strategy is used. Network is OF features. Most imperative and prognostic retinal region for
trained through stochastic gradient descent and Nestrov mo- AMD diagnosis is the central part of retina. This information is
mentum weight learning methods. After pre-training, network is utilized by extracting features appended from multiple concentric
further trained with real time augmented image patches. This square grids. LSVM is trained with extracted features. Efficiency
is done to improve localization ability of system. Legitimacy of of model is scrutinized using NIH AREDS dataset. The dataset
algorithm is tested on Kaggle dataset. Proposed method provided is divided into two classes i.e. EIPC (equal number of images
95% sensitivity and 75% accuracy as presented in Table 8. per class) and MIPC (maximum number of images per class).
Abràmoff et al. [72] provided an improved version of their Performance is evaluated on both classes of dataset as presented
previous work for classification of DR and ME, older version of in Table 8.
algorithm was developed without incorporating deep learning. Burlina et al. [77] have extended their work to identify the
New algorithm provides automatic identification and Classifica- class of AMD. Class 1 corresponds to no AMD, class 2 incorporates
tion of DR into moderate, severe non proliferative DR (NPDR), early stage AMD cases, class 3 encompasses intermediate stage
proliferative DR (PDR), and ME with improved statistical mea- AMD, and class 4 denotes advanced form of AMD. They have used
sures. They used IDX-DR version X2.1 automated system. The OverFeat deep CNN for finding the severity grade of AMD. The
device was trained using EyeCheck project dataset. Evaluation of efficacy of the algorithm is scrutinized using NIH AREDS dataset.
system is done using Messidor-2 dataset. The statistical findings The obtained performance measures are shown in Table 8.
are presented in Table 8. 3-D OCT imaging modality is the most common among imag-
A method for identification of DR and diabetic ME from reti- ing modalities in the field of ophthalmology. OCT images com-
nal fundus images has been provided by Gulshan et al. [64]. bined with electronic medical records (EMR) give a rich dataset
They used Inception v3 DNN architecture [89]. EyePACS-1 and for training of DNNs. Lee et al. [78] have followed the same
MESSIDOR-2 datasets are used for training and testing of net- footings for AMD detection. They implemented VGG-16 DNN for
work. ImageNet dataset is used for initialization of network efficient classification of AMD using OCT images. Their network
weights. The weights are learned via distributed stochastic gra- consists of 21 layers, including convolutional layers and max
dient and training of network is made efficient by using batch pooling layers, each with a ReLU activation unit. Automatically
 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203 15

Table 7
Performance measures for Simultaneous Segmentation of Retinal anatomical structures using Supervised learning Methods.
Dataset Class SN SP Acc AUC
Background 95.47% 80.63% – –
DRIVE OD 87.90% 99.27% – –
Tan et al. [66]
Fovea 88.53% 99.14% – –
Blood Vessels 75.73% 96.94% – –
Hemorrhages 62.57% 98.93% – –
CLEOPATRA
Tan et al. [68] Hard Exudates 87.58% 98.73% – –
Microaneurysms 46.06% 97.99% – –
Hemorrhages 80.93% 98.54% 97.86% –
Messidor
Badar et al. [70] Hard Exudates 90.69% 99.64% 99.24% –
Soft Exudates 72.87% 89.32% 88.65% –
Hard Exudates – – – 95%
Lam et al. [69] E-Ophta
Microaneurysms – – – 94%

Table 8
Performance measures for Retinal Disease classification using Supervised/Unsupervised learning methods.
Dataset SN SP Acc AUC Method Focus
Colas et al. [73] Kaggle DR 96.2% 66.6% – 94.6%
detection
dataset
Pratt et al. [71] Kaggle DR 95% – 75% –
detection
dataset
DR Identification/
Abràmoff et al. MESSIDOR-2 96.8% 87% – 98% Supervised
Classification
[72]
Gulshan et al. EyePACS-1 90.3% 98.1% – –
[64] MESSIDOR-2 87% 98.5% – –
Gargeya and MESSIDOR 2 93% 87% – 94%
Leng [74] E-OPTHA 90% 94% – 95%
Worrall et al. Private dataset 93.6% 95.4% 94.7% – Supervised ROP Identification
[75]
Burlina et al. NIH AREDS 90.9%–95.7% 90.1%–95.6% 91.9%–95% –
[76]
Lee et al. [78] 3-D OCT 92.64% 93.69% 93.45% 97.45% AMD Identification/
Supervised
Images Classification
Burlina et al. – – 85% –
[77]
Choi et al. [79] STARE 36.7% Supervised
Arunkumar and ARIA 79.32% 97.89% 87.62% – Unsupervised Multiple Retinal
Karthigaikumar diseases classification
[3]

extracted OCT images are used for training and testing of net- [3]. They used Generalized Regression Neural Network (GRNN) for
work. Weights are initialized using Xavier algorithm [86] and reduction of feature vector dimension. This is done to improve
optimized by stochastic gradient descent. Input images are first compute time efficiency. The model is able to extract intricate
downsized and histogram equalized and are then fed into the features because it includes stacked Restricted Boltzmann Ma-
network. Results are shown in Table 8. chine (RBM) [90] in its layers. Input images are first preprocessed
Choi et al. [79] proposed a deep CNN for systematic detection to remove noise and adjust contrast. Effectiveness of system is
of multiple retinal diseases using fundus photographs of STARE checked on ARIA dataset. Results are recorded in Table 8.
image set. The data set was expanded by incorporating nine more
diseases such background DR, PDR, Dry AMD, Wet AMD, Retinal 5. Discussion
Vein Occlusion, Retinal Artery Occlusion, Hypertensive retinopa-
thy, Coat’s disease, and Retinitis. Initially, random forest with From our survey of DNNs for retinal image analysis we can de-
VGG19 transfer learning method is used to classify the diseases duce that overall supervised learning approaches have performed
but due to the multiple diseases, accuracy was quite low. Later better as compared to unsupervised learning methods because
ensemble classifiers are introduced to achieve better classification supervised learning networks learn the mapping efficiently due
accuracy of multiple diseases. The network is reported to achieve to the presence of ground truth data. Most of the progress in
36.7% accuracy in the classification task of nine retinal diseases. deep learning based retinal image analysis has been witnessed
in the segmentation of retinal vasculature. Among all the DNN
4.5.2. Unsupervised methods based algorithms which utilize intensity level pixel information,
Unsupervised DNNs have proved effective for classification of [55] have provided the maximum AUC of 98.79%. However, use
retinal diseases such as: AMD, DR, Macular Bunker, Retinoblas- of only intensity level information undermines the efficiency
toma, Retinal Detachment, and Retinitis Pigmentosa. This ap- of algorithm because neighboring pixels always have a certain
proach has been suggested by Arunkumar and Karthigaikumar correlation factor. The blood vessel segmentation methodology
16 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

proposed by Liskowski and Krawiec [54] have utilized contex- [4] A. Sopharak, B. Uyyanonvara, S. Barman, T.H. Williamson, Automatic
tual information along with intensity level information of pixels detection of diabetic retinopathy exudates from non-dilated retinal images
using mathematical morphology methods, Comput. Med. Imaging Graph.
for blood vessel segmentation. Because of the use of neighbor-
32 (8) (2008) 720–727.
hood information, their model outperformed other retinal blood [5] T. Walter, J.-C. Klein, P. Massin, A. Erginay, A contribution of image
vessel segmentation techniques. It achieved maximum AUC of processing to the diagnosis of diabetic retinopathy-detection of exudates
99.28% which is higher than best to date AUC recorded by other in color fundus images of the human retina, IEEE Trans. Med. Imaging 21
(10) (2002) 1236–1243.
traditional approaches. Instead of following a tedious approach
[6] M.M. Fraz, P. Remagnino, A. Hoppe, B. Uyyanonvara, A.R. Rudnicka, C.G.
i.e. one by one retinal landmark detection, DNNs can be used for Owen, S.A. Barman, An ensemble classification-based approach applied to
simultaneous extraction of retinal landmarks from retinal images. retinal blood vessel segmentation, IEEE Trans. Biomed. Eng. 59 (9) (2012)
This approach is presented by Tan et al. [66]. DNN have not yet 2538–2548.
been much explored for retinal pathology detection. Prentašić and [7] G. Quellec, M. Lamard, P.M. Josselin, G. Cazuguel, B. Cochener, C. Roux,
Optimal wavelet transform for the detection of microaneurysms in retina
Lončarić [91] have provided state-of-the-art results for detection photographs, IEEE Trans. Med. Imaging 27 (9) (2008) 1230–1241.
of exudates. The method proposed by Shan and Li [92] for mi- [8] D. Marín, A. Aquino, M.E. Gegúndez-Arias, J.M. Bravo, A new supervised
croaneurysms detection through SSAE achieved an AUC of 96.20%. method for blood vessel segmentation in retinal images by using gray-
Some of the recent works have explored supervised DNNs for level and moment invariants-based features, IEEE Trans. Med. Imaging 30
(1) (2011) 146–158.
simultaneous segmentation of retinal pathologies. Tan et al. [68] [9] Y. Kanagasingam, A. Bhuiyan, M.D. Abràmoff, R.T. Smith, L. Goldschmidt,
provided a segmentation network for simultaneous segmentation T.Y. Wong, Progress on retinal image analysis for age related macular
of exudates, hemorrhages, and microaneurysms. Badar et al. [70] degeneration, Progress Retin. Eye Res. 38 (2014) 20–42.
presented a better approach by providing a method for pixel level [10] I. Sadek, D. Sidibé, F. Meriaudeau, Automatic discrimination of color retinal
images using the bag of words approach, in: Paper presented at the SPIE
segmentation of exudates, hemorrhages, and cotton wool spots.
Medical Imaging, 2015.
Deep learning is still an underexplored technique in the field of [11] D. Sidibé, I. Sadek, F. Mériaudeau, Discrimination of retinal images con-
retinal pathology detection. Lee et al. [78] have proved efficacy of taining bright lesions using sparse coded features and SVM, Comput. Biol.
DNN for grading of AMD in 3-D OCT images by recording an AUC Med. 62 (2015) 175–184.
[12] R. Veras, R. Silva, F. Araújo, F. Medeiros, SURF descriptor and pattern
of 97.45%. Supervised DNN approach have also been employed for
recognition techniques in automatic identification of pathological retinas,
efficient grading of DR [64,71,73,74] are examples of this effort. in: Paper Presented at the Intelligent Systems (BRACIS), 2015 Brazilian
Among all of them, [72] have achieved the maximum AUC of Conference on, 2015.
98.0% for grading of DR. [13] N. Patton, T.M. Aslam, T. MacGillivray, I.J. Deary, B. Dhillon, R.H. Eikelboom,
et al., Retinal image analysis: Concepts, applications and potential, Progress
Retin. Eye Res. 25 (1) (2006) 99–127.
6. Conclusion [14] D.M. Albert, W.H. Miller, Jan purkinje and the ophthalmoscope, Am. J.
Ophthalmol. 76 (4) (1973) 494–499.
Retinal image analysis through DNNs is a nascent field. Al- [15] D. Huang, E.A. Swanson, C.P. Lin, J.S. Schuman, W.G. Stinson, W. Chang,
et al., Optical coherence tomography, Science 254 (5035) (1991) 1178.
though research has been conducted in extraction of retinal land-
[16] T.J. Bennett, C.J. Barry, Ophthalmic imaging today: An ophthalmic pho-
marks and pathologies but the epitome of this technique is yet tographer’s viewpoint–a review, Clin. Exp. Ophthalmol. 37 (1) (2009)
to be witnessed. However, unsupervised learning based DNNs 2–13.
have not seen much progress in retinal image analysis. Deep [17] P. Venkatesh, R. Sharma, N. Vashist, R. Vohra, S. Garg, Detection of retinal
lesions in diabetic retinopathy: Comparative evaluation of 7-field digital
learning techniques can be efficiently applied for segmentation
color photography versus red-free photography, Int. Ophthal. 35 (5) (2015)
of dot and blot hemorrhages, cotton wool spots, hard exudates, 635–640.
soft exudates, drusen etc. There is no restriction on number of [18] M.E. Tyler, Stereo fundus photography: Principles and technique, J.
layers and architecture of neural network, network architecture Ophthalmic Photogr. 18 (2) (1996) 6–81.
is chosen heuristically in accordance with problem domain. The [19] Y. Hirohara, Y. OKawa, T. Mihashi, T. Yamaguchi, N. Nakazawa, Y. Tsuruga,
et al., Validity of retinal oxygen saturation analysis: Hyperspectral imaging
variants of Deep Neural Networks like AlexNet, LSTM, VggNet, in visible wavelength with fundus camera and liquid crystal wavelength
and GoogleNet can be used for extraction of retinal anatomical tunable filter, Opt. Rev. 14 (3) (2007) 151.
structures. Although VGG-16 is used by Lee et al. [78] for 3-D [20] I. Alabboud, G. Muyo, A. Gorman, D. Mordant, A. McNaught, C. Petres,
OCT retinal image analysis; however, there is no precedence of et al., New spectral imaging techniques for blood oximetry in the retina, in:
Paper Presented at the European Conference on Biomedical Optics, 2007.
its use in case of color fundus images. All these networks are very [21] R.H. Webb, G.W. Hughes, Scanning laser ophthalmoscope, IEEE Trans.
deep and they have capability of extracting much more complex Biomed. Eng. (7) (1981) 488–492.
features than those extracted by traditional methods and with [22] A. Roorda, F. Romero-Borja, W.J. Donnelly I.I.I., H. Queener, T.J. Hebert, M.C.
better performance measures. This property makes DNN capable Campbell, Adaptive optics scanning laser ophthalmoscopy, Opt. Exp. 10 (9)
(2002) 405–412.
of replacing traditional ophthalmologic screening practices.
[23] E. Ng, U.R. Acharya, R.M. Rangayyan, J.S. Suri, Ophthalmological Imaging
and Applications, CRC Press, 2014.
Declaration of competing interest [24] J.S. Slakter, L.A. Yannuzzi, D.R. Guyer, J.A. Sorenson, D.A. Orlock,
Indocyanine-green angiography, Curr. Opin. Ophthalmol. 6 (3) (1995)
25–32.
The authors declare that they have no known competing finan- [25] W. Drexler, J.G. Fujimoto, Optical Coherence Tomography: Technology and
cial interests or personal relationships that could have appeared Applications, Springer, 2015.
to influence the work reported in this paper. [26] F.A. Jakobiec, Ocular Anatomy, Embryology, and Teratology, Harpercollins,
1982.
[27] J. Beagley, L. Guariguata, C. Weil, A.A. Motala, Global estimates of un-
References diagnosed diabetes in adults, Diabetes Res. Clin. Pract. 103 (2) (2014)
150–160.
[1] M.M. Fraz, P. Remagnino, A. Hoppe, B. Uyyanonvara, A.R. Rudnicka, C.G. [28] V. Raman, P. Then, P. Sumari, Proposed retinal abnormality detection and
Owen, S.A. Barman, Blood vessel segmentation methodologies in retinal classification approach: Computer aided detection for diabetic retinopathy
images–a survey, Comput. Methods Programs Biomed. 108 (1) (2012) by machine learning approaches, in: Paper Presented at the Commu-
407–433. nication Software and Networks (ICCSN), 2016 8th IEEE International
[2] M.D. Abràmoff, M.K. Garvin, M. Sonka, Retinal imaging and image analysis, Conference on, 2016.
IEEE Rev. Biomed. Eng. 3 (2010) 169–208. [29] R.L. Engerman, Pathogenesis of diabetic retinopathy, Diabetes 38 (10)
[3] R. Arunkumar, P. Karthigaikumar, Multi-retinal disease classification by (1989) 1203–1206.
reduced deep learning features, Neural Comput. Appl. 28 (2) (2017) [30] R.A. Kowluru, P.-S. Chan, Capillary Dropout in Diabetic Retinopathy
329–334, http://dx.doi.org/10.1007/s00521-015-2059-9. Diabetic Retinopathy, Springer, 2008, pp. 265–282.
 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203 17

[31] M. Niemeijer, J. Staal, B. Ginneken, M. Loog, M. Abramoff, DRIVE: Digital [58] R. Welikala, P. Foster, P. Whincup, A. Rudnicka, C. Owen, D. Strachan, S.
retinal images for vessel extraction, in: Methods for Evaluating Seg- Barman, Automated arteriole and venule classification using deep learning
mentation and Indexing Techniques Dedicated To Retinal Ophthalmology, for retinal images from the UK Biobank cohort, Comput. Biol. Med. (2017).
2004. [59] X. Xu, T. Tan, F. Xu, An improved U-net architecture for simultaneous
[32] A. Hoover, V. Kouznetsova, M. Goldbaum, Locating blood vessels in retinal arteriole and venule segmentation in fundus image, in: Paper presented
images by piecewise threshold probing of a matched filter response, IEEE at the Annual Conference on Medical Image Understanding and Analysis,
Trans. Med. Imaging 19 (3) (2000) 203–210. 2018.
[33] ARIA Online, Retinal image archive, 2006, http://www.eyecharity.com/aria [60] D. Maji, A. Santara, S. Ghosh, D. Sheet, P. Mitra, Deep neural network and
online/. random forest hybrid architecture for learning to detect retinal vessels in
[34] T. Kauppi, V. Kalesnykiene, J.-K. Kamarainen, L. Lensu, I. Sorri, H. Uusitalo, fundus images, in: Paper Presented at the Engineering in Medicine and
et al., DIARETDB0: Evaluation database and methodology for diabetic Biology Society (EMBC), 2015 37th Annual International Conference of the
retinopathy algorithms, in: Machine Vision and Pattern Recognition Re- IEEE, 2015.
search Group, Lappeenranta University of Technology, Finland, 2006, p. [61] A. Lahiri, A.G. Roy, D. Sheet, P.K. Biswas, Deep neural ensemble for
73. retinal vessel segmentation in fundus images towards achieving label-
[35] R. Kälviäinen, H. Uusitalo, DIARETDB1 diabetic retinopathy database free angiography, in: Paper presented at the Engineering in Medicine and
and evaluation protocol, in: Paper Presented at the Medical Image Biology Society (EMBC), 2016 IEEE 38th Annual International Conference
Understanding and Analysis, 2007. of the, 2016.
[36] E. Decencière, X. Zhang, G. Cazuguel, B. Laÿ, B. Cochener, C. Trone, et al., [62] P. Prentašić, S. Lončarić, Detection of exudates in fundus photographs using
Feedback on a publicly distributed image database: The Messidor database, deep neural networks and anatomical landmark detection fusion, Comput.
Image Anal. Stereol. 33 (3) (2014) 231–234. Methods Programs Biomed. 137 (2016) 281–292.
[37] P. Prentašić, S. Lončarić, Z. Vatavuk, G. Benčić, M. Subašić, T. Petković, et al.,
[63] S. Abbasi-Sureshjani, B. Dashtbozorg, B.M. ter Haar Romeny, F. Fleuret,
Diabetic retinopathy image database(DRiDB): A new database for diabetic
Boosted exudate segmentation in retinal images using residual nets, in:
retinopathy screening programs research, in: Paper Presented at the 2013
Fetal, Infant and Ophthalmic Medical Image Analysis, Springer, 2017, pp.
8th International Symposium on Image and Signal Processing and Analysis,
210–218.
ISPA, 2013.
[64] V. Gulshan, L. Peng, M. Coram, M.C. Stumpe, D. Wu, A. Narayanaswamy,
[38] J.J. Kanski, B. Bowling, Clinical Ophthalmology: A Systematic Approach,
et al., Development and validation of a deep learning algorithm for
Elsevier Health Sciences, 2011.
detection of diabetic retinopathy in retinal fundus photographs, JAMA 316
[39] R. Klein, M.D. Knudtson, K.E. Lee, R. Gangnon, B.E. Klein, The Wisconsin
(22) (2016) 2402–2410.
Epidemiologic Study of Diabetic Retinopathy XXIII: The twenty-five-
year incidence of macular edema in persons with type 1 diabetes, [65] K.-K. Maninis, J. Pont-Tuset, P. Arbeláez, L. Van Gool, Deep retinal image
Ophthalmology 116 (3) (2009) 497–503. understanding, in: Paper Presented at the International Conference on
[40] R. Varma, N.M. Bressler, Q.V. Doan, M. Gleeson, M. Danese, J.K. Bower, Medical Image Computing and Computer-Assisted Intervention, 2016.
et al., Prevalence of and risk factors for diabetic macular edema in the [66] J.H. Tan, U.R. Acharya, S.V. Bhandary, K.C. Chua, S. Sivaprasad, Segmentation
United States, JAMA Ophthalmol. 132 (11) (2014) 1334–1340. of optic disc, fovea and retinal vasculature using a single convolutional
[41] R.D. Jager, W.F. Mieler, J.W. Miller, Age-related macular degeneration, N. neural network, J. Comput. Sci. (2017).
Engl. J. Med. 358 (24) (2008) 2606–2617. [67] J. Zilly, J.M. Buhmann, D. Mahapatra, Glaucoma detection using en-
[42] D.V. Alfaro, Age-Related Macular Degeneration: A Comprehensive tropy sampling and ensemble learning for automatic optic cup and disc
Textbook, Lippincott Williams & Wilkins, 2006. segmentation, Comput. Med. Imaging Graph. 55 (2017) 28–41.
[43] G. Liew, J.J. Wang, Retinal vascular signs: A window to the heart? Rev. [68] J.H. Tan, H. Fujita, S. Sivaprasad, S.V. Bhandary, A.K. Rao, K.C. Chua, U.R.
Esp. Cardiol. (English Edition) 64 (6) (2011) 515–521. Acharya, Automated segmentation of exudates, haemorrhages, microa-
[44] H.A. Quigley, A.T. Broman, The number of people with glaucoma worldwide neurysms using single convolutional neural network, Inform. Sci. 420
in 2010 and 2020, Br. J. Ophthalmol. 90 (3) (2006) 262–267. (2017) 66–76.
[45] R. Thomas, R.S. Parikh, How to assess a patient for glaucoma, Community [69] C. Lam, C. Yu, L. Huang, D. Rubin, Retinal lesion detection with deep
Eye Health 19 (59) (2006) 36. learning using image patches, Invest. Ophthalmol. Vis. Sci. 59 (1) (2018)
[46] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT press, 2016. 590–596.
[47] F.-H. Hsu, Behind Deep Blue: Building the Computer that Defeated the [70] M. Badar, M. Shahzad, M. Fraz, Simultaneous segmentation of multiple
World Chess Champion, Princeton University Press, 2002, http://swinghu. retinal pathologies using fully convolutional deep neural network, in: Pa-
github.io/deep_learning/2015/10/09/dl-tutorials.html. per Presented at the Annual Conference on Medical Image Understanding
[48] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015) and Analysis, 2018.
436–444. [71] H. Pratt, F. Coenen, D.M. Broadbent, S.P. Harding, Y. Zheng, Convolutional
[49] P. Angelov, A. Sperduti, Challenges in deep learning, in: Paper Presented neural networks for diabetic retinopathy, Procedia Comput. Sci. 90 (2016)
at the Proceedings of the 24th European symposium on artificial neural 200–205.
networks, ESANN, 2016. [72] M.D. Abràmoff, Y. Lou, A. Erginay, W. Clarida, R. Amelon, J.C. Folk, M.
[50] S. Wang, Y. Yin, G. Cao, B. Wei, Y. Zheng, G. Yang, Hierarchical reti- Niemeijer, Improved automated detection of diabetic retinopathy on a
nal blood vessel segmentation based on feature and ensemble learning, publicly available dataset through integration of deep learningdeep learn-
Neurocomputing 149 (2015) 708–717. ing detection of diabetic retinopathy, Invest. Ophthalmol. Vis. Sci. 57 (13)
[51] T. Fang, R. Su, L. Xie, Q. Gu, Q. Li, P. Liang, T. Wang, Retinal vessel landmark
(2016) 5200–5206.
detection using deep learning and hessian matrix, in: Paper Presented at
[73] E. Colas, A. Besse, A. Orgogozo, B. Schmauch, N. Meric, E. Besse, Deep
the Image and Signal Processing (CISP), 2015 8th International Congress
learning approach for diabetic retinopathy screening, Acta Ophthalmol. 94
on, 2015.
(S256) (2016).
[52] M. Melinščak, P. Prentašić, S. Lončarić, Retinal vessel segmentation using
[74] R. Gargeya, T. Leng, Automated identification of diabetic retinopathy using
deep neural networks, in: Paper Presented at the VISAPP 2015 (10th
deep learning, Ophthalmology (2017).
International Conference on Computer Vision Theory and Applications),
2015. [75] D.E. Worrall, C.M. Wilson, G.J. Brostow, Automated retinopathy of prematu-
[53] H. Fu, Y. Xu, S. Lin, D.W.K. Wong, J. Liu, DeepVessel: Retinal vessel rity case detection with convolutional neural networks, in: Deep Learning
segmentation via deep learning and conditional random field, in: Paper and Data Labeling for Medical Applications, Springer, 2016, pp. 68–76.
presented at the International Conference on Medical Image Computing [76] P. Burlina, D.E. Freund, N. Joshi, Y. Wolfson, N.M. Bressler, Detection of
and Computer-Assisted Intervention, 2016. age-related macular degeneration via deep learning, in: Paper Presented
[54] P. Liskowski, K. Krawiec, Segmenting retinal blood vessels with<? pub at the Biomedical Imaging (ISBI), 2016 IEEE 13th International Symposium
_newline?> deep neural networks, IEEE Trans. Med. Imaging 35 (11) (2016) on, 2016.
2369–2380. [77] P. Burlina, K.D. Pacheco, N. Joshi, D.E. Freund, N.M. Bressler, Comparing
[55] Q. Li, B. Feng, L. Xie, P. Liang, H. Zhang, T. Wang, A cross-modality learning humans and deep learning performance for grading AMD: A study in using
approach for vessel segmentation in retinal images, IEEE Trans. Med. universal deep features and transfer learning for automated AMD analysis,
Imaging 35 (1) (2016) 109–118. Comput. Biol. Med. 82 (2017) 80–86.
[56] Z. Yao, Z. Zhang, L.-Q. Xu, Convolutional neural network for retinal blood [78] C.S. Lee, D.M. Baughman, A.Y. Lee, Deep learning is effective for classi-
vessel segmentation, in: Paper Presented at the Computational Intelligence fying normal versus age-related macular degeneration optical coherence
and Design (ISCID), 2016 9th International Symposium on, 2016. tomography images, Ophthalmol. Retin. (2017) http://dx.doi.org/10.1016/j.
[57] A. Dasgupta, S. Singh, A fully convolutional neural network based struc- oret.2016.12.009.
tured prediction approach towards the retinal vessel segmentation, in: [79] J.Y. Choi, T.K. Yoo, J.G. Seo, J. Kwak, T.T. Um, T.H. Rim, Multi-categorical
Paper Presented at the Biomedical Imaging (ISBI 2017), 2017 IEEE 14th deep learning neural network to classify retinal images: A pilot study
International Symposium on, 2017. employing small database, PLoS One 12 (11) (2017) e0187336.
18 M. Badar, M. Haris and A. Fatima / Computer Science Review 35 (2020) 100203

[80] S. Xie, Z. Tu, Holistically-nested edge detection, in: Paper Presented at [87] J. Sivaswamy, S. Krishnadas, G.D. Joshi, M. Jain, A.U.S. Tabish, Drishti-gs:
the Proceedings of the IEEE International Conference on Computer Vision, Retinal image dataset for optic nerve head (onh) segmentation, in: Paper
2015. Presented at the 2014 IEEE 11th International Symposium on Biomedical
[81] T. Walter, J.-C. Klein, Segmentation of color fundus images of the human Imaging (ISBI), 2014.
retina: Detection of the optic disc and the vascular tree using morpho- [88] V. Badrinarayanan, A. Kendall, R. Cipolla, Segnet: A deep convolutional
logical techniques, in: Paper Presented at the International Symposium on encoder–decoder architecture for image segmentation, IEEE Trans. Pattern
Medical Data Analysis, 2001. Anal. Mach. Intell. 39 (12) (2017) 2481–2495.
[82] A.F. Frangi, W.J. Niessen, K.L. Vincken, M.A. Viergever, Multiscale vessel [89] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the
enhancement filtering, in: Paper Presented at the International Conference inception architecture for computer vision, in: Paper Presented at the
on Medical Image Computing and Computer-Assisted Intervention, 1998. Proceedings of the IEEE Conference on Computer Vision and Pattern
[83] P. Costa, A. Campilho, Convolutional bag of words for diabetic retinopathy Recognition, 2016.
detection from eye fundus images, IPSJ Trans. Comput. Vis. Appl. 9 (1) [90] G. Dahl, A.-r. Mohamed, G.E. Hinton, Phone recognition with the mean-
(2017) 10, http://dx.doi.org/10.1186/s41074-017-0023-6. covariance restricted Boltzmann machine, in: Paper Presented at the
[84] S. Otálora, O. Perdomo, F. González, H. Müller, Training deep convolutional Advances in Neural Information Processing Systems, 2010.
neural networks with active learning for exudate classification in eye [91] P. Prentašić, S. Lončarić, Detection of exudates in fundus photographs
fundus images, in: Intravascular Imaging and Computer Assisted Stenting, using convolutional neural networks, in: Paper presented at the Image and
and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, Signal Processing and Analysis (ISPA), 2015 9th International Symposium
Springer, 2017, pp. 146–154. on, 2015.
[85] Y. Nesterov, A method for unconstrained convex minimization problem [92] J. Shan, L. Li, A deep learning method for microaneurysm detection in
with the rate of convergence O (1/k^2), in: Paper Presented at the Doklady fundus images, in: Paper presented at the Connected Health: Applica-
AN USSR, 1983. tions, Systems and Engineering Technologies (CHASE), 2016 IEEE First
[86] X. Glorot, Y. Bengio, Understanding the difficulty of training deep feed- International Conference on, 2016.
forward neural networks, in: Paper Presented at the Proceedings of the
Thirteenth International Conference on Artificial Intelligence and Statistics,
2010.