Review

Br J Ophthalmol: first published as 10.1136/bjophthalmol-2018-313173 on 25 October 2018. Downloaded from http://bjo.bmj.com/ on October 21, 2022 at India:BMJ-PG Sponsored.
Artificial intelligence and deep learning
in ophthalmology
Daniel Shu Wei Ting,1 Louis R Pasquale,2 Lily Peng,3 John Peter Campbell,4
Aaron Y Lee,5 Rajiv Raman,6 Gavin Siew Wei Tan,1 Leopold Schmetterer,1,7,8,9
Pearse A Keane,10 Tien Yin Wong1

For numbered affiliations see Abstract ocular imaging, principally fundus photographs
end of article. Artificial intelligence (AI) based on deep learning (DL) and optical coherence tomography (OCT). Major
has sparked tremendous global interest in recent years. ophthalmic diseases which DL techniques have been
Correspondence to used for include diabetic retinopathy (DR),11–15
DL has been widely adopted in image recognition,
Dr Daniel Shu Wei Ting,
Assistant Professor in speech recognition and natural language processing, glaucoma,11 16 age-related macular degeneration
Ophthalmology, Duke-NUS but is only beginning to impact on healthcare. In (AMD)11 17 18 and retinopathy of prematurity
Medical SchoolSingapore ophthalmology, DL has been applied to fundus (ROP).19 DL has also been applied to estimate
National Eye Center, Singapore photographs, optical coherence tomography and visual refractive error and cardiovascular risk factors (eg,
168751, Singapore; ​daniel.​ting.​
s.​w@​singhealth.c​ om.​sg fields, achieving robust classification performance in age, blood pressure, smoking status and body mass
the detection of diabetic retinopathy and retinopathy of index).20 21
Received 4 September 2018 prematurity, the glaucoma-like disc, macular oedema and A primary benefit of DL in ophthalmology could
Revised 17 September 2018 age-related macular degeneration. DL in ocular imaging be in screening, such as for DR and ROP, for which
Accepted 23 September 2018 well-established guidelines exist. Other conditions,
Published Online First
may be used in conjunction with telemedicine as a
25 October 2018 possible solution to screen, diagnose and monitor major such as glaucoma and AMD, may also require
eye diseases for patients in primary care and community screening and long-term follow-up. However,
settings. Nonetheless, there are also potential challenges screening requires tremendous manpower and
financial resources from healthcare systems, in

Protected by copyright.
with DL application in ophthalmology, including clinical
and technical challenges, explainability of the algorithm both developed countries and in low-income and
results, medicolegal issues, and physician and patient middle-income countries. The use of DL, coupled
acceptance of the AI ’black-box’ algorithms. DL could with telemedicine, may be a long-term solution
potentially revolutionise how ophthalmology is practised to screen and monitor patients within primary
in the future. This review provides a summary of the eye care settings. This review summarises new DL
state-of-the-art DL systems described for ophthalmic systems for ophthalmology applications, potential
applications, potential challenges in clinical deployment challenges in clinical deployment and potential
and the path forward. paths forward.

DL applications in ophthalmology
Diabetic retinopathy
Introduction Globally, 600 million people will have diabetes by
Artificial intelligence (AI) is the fourth industrial 2040, with a third having DR.22 A pooled analysis
revolution in mankind’s history.1 Deep learning of 22 896 people with diabetes from 35 popula-
(DL) is a class of state-of-the-art machine learning tion-based studies in the USA, Australia, Europe
techniques that has sparked tremendous global and Asia (between 1980 and 2008) showed that the
interest in the last few years.2 DL uses represen- overall prevalence of any DR (in type 1 and type
tation-learning methods with multiple levels of 2 diabetes) was 34.6%, with 7% vision-threatening
abstraction to process input data without the need diabetic retinopathy.22 Screening for DR, coupled
for manual feature engineering, automatically with timely referral and treatment, is a universally
recognising the intricate structures in high-di- accepted strategy for blindness prevention. DR
mensional data through projection onto a lower screening can be performed by different health-
dimensional manifold.2 Compared with conven- care professionals, including ophthalmologists,
tional techniques, DL has been shown to achieve optometrists, general practitioners, screening tech-
significantly higher accuracies in many domains, nicians and clinical photographers. The screening
including natural language processing, computer methods comprise direct ophthalmoscopy,23 dilated
© Author(s) (or their vision3–5 and voice recognition.6 slit lamp biomicroscopy with a hand-held lens (90
employer(s)) 2019. Re-use
permitted under CC BY-NC. No In medicine and healthcare, DL has been primarily D or 78 D),24 mydriatic or non-mydriatic retinal
commercial re-use. See rights applied to medical imaging analysis, in which DL photography,23 teleretinal screening,25 and retinal
and permissions. Published systems have shown robust diagnostic perfor- video recording.26 Nonetheless, DR screening
by BMJ. mance in detecting various medical conditions, programmes are challenged by issues related to
To cite: Ting DSW, including tuberculosis from chest X-rays,7 8 malig- implementation, availability of human assessors and
Pasquale LR, Peng L, et al. nant melanoma on skin photographs9 and lymph long-term financial sustainability.27
Br J Ophthalmol node metastases secondary to breast cancer from Over the past few years, DL has revolutionised
2019;103:167–175. tissue sections.10 DL has similarly been applied to the diagnostic performance in detecting DR.2 Using
Ting DSW, et al. Br J Ophthalmol 2019;103:167–175. doi:10.1136/bjophthalmol-2018-313173 167
 Review

Br J Ophthalmol: first published as 10.1136/bjophthalmol-2018-313173 on 25 October 2018. Downloaded from http://bjo.bmj.com/ on October 21, 2022 at India:BMJ-PG Sponsored.
Table 1  Summary table for the different DL systems in the detection of referable diabetic retinopathy, glaucoma suspect, age-related macular
degeneration and retinopathy of prematurity using fundus photographs
DL systems Year Test data sets Test images (n) CNN AUC Sensitivity (%) Specificity (%)
Referable diabetic retinopathy
 Abràmoff et al14 2016 Messidor-2 1748 AlexNet/VGG 0.98 96.80 87.00
 Gulshan et al12 2016 Messidor-2 1748 Inception-V3 0.99 87 98.50
96.10 93.90
EyePACS-1 9963 0.991 90.30 98.10
97.50 93.40
 Gargeya and Leng15 2017 Kaggle images 75 137 Customised CNN 0.97 NA NA
  E-Ophtha 463 0.96 NA NA
  Messidor-2 1748 0.94 NA NA
 Ting et al11 2017 SiDRP 14–15 71 896 VGG-19 0.936 90.50 91.60
  Guangdong 15 798 0.949 98.70 81.60
  SIMES 3052 0.889 97.10 82.00
  SINDI 4512 0.917 99.3 73.3
  SCES 1936 0.919 100 76.30
  BES 1052 0.929 94.40 88.50
  AFEDS 1968 0.98 98.80 86.50
  RVEEH 2302 0.983 98.90 92.20
  Mexican 1172 0.95 91.80 84.80
  CUHK 1254 0.948 99.3 83.10
  HKU 7706 0.964 100 81.30
 Abràmoff et al28 2018 10 primary care practice 892 patients Alex/VGG NA 87.2 90.7
sites from the USA
Glaucoma suspect*

Protected by copyright.
 Ting et al11 2017 SiDRP 14–15 71 896 VGG-19 0.942 96.40 93.20
 Li et al16 2018 Guangdong 48 116 0.986 95.60 92.00
Age-related macular degeneration
 Ting et al11 2017 SiDRP 14–15 35 948 VGG-19 0.932 93.20 88.70
 Burlina et al17 2017 AREDS 120 656 AlexNet, OverFeat 0.940–0.96 NA NA
 Grassmann et al18 2018 AREDS 120 656 AlexNet, GoogleNet, VGG, NA 84.20 94.30
Inception-V3, ResNet, Inception-
ResNet-V2
Retinopathy of prematurity
 Brown et al19 2018 i-ROP 100 Inception-V1 and U-Net NA 100 94
The diagnostic performance is not comparable between the different DL systems given the different data sets used in the individual study.
*Definition of glaucoma suspect: (1) Ting et al11—vertical cup to disc ratio of 0.8 or greater, and any glaucomatous disc changes; (2) Li et al16—vertical cup to disc ratio of 0.7 or
greater, and any glaucomatous disc changes.
AFEDS, African American Eye Disease Study; AREDS, Age-Related Eye Disease Study; AUC, area under the receiver operating characteristic curve; BES, Beijing Eye Study; CNN,
convolutional neural network; CUHK, Chinese University Hong Kong; DL, deep learning; SiDRP 14–15, Singapore Integrated Diabetic Retinopathy Screening Programme; HKU,
Hong Kong University; NA, not available; RVEEH, Royal Victorian Eye and Ear Hospital; SCES, Singapore Chinese Eye Study; SIMES, Singapore Malay Eye Study; SINDI, Singapore
Indian Eye Study.

this technique, many groups have shown excellent diagnostic graded by at least seven US board-certified ophthalmologists
performance (table 1).14 Abràmoff et al14 showed that a DL with high intragrader consistency. The AUC was 0.991 and
system was able to achieve an area under the receiver operating 0.990 for EyePACS-1 and Messidor-2, respectively (table 1).
characteristic curve (AUC) of 0.980, with sensitivity and speci- Although a number of groups have demonstrated good results
ficity of 96.8% and 87.0%, respectively, in the detection of refer- using DL systems on publicly available data sets, the DL systems
able DR (defined as moderate non-proliferative DR or worse, were not tested in real-world DR screening programmes. In
including diabetic macular oedema (DMO)) on Messidor-2 data addition, the generalisability of a DL system to populations of
set. Similarly, Gargeya and Leng15 reported an AUC of 0.97 different ethnicities, and retinal images captured using different
using cross-validation on the same data set, and 0.94 and 0.95 in cameras, still remains uncertain. Ting et al11 reported a clinically
two independent test sets (Messidor-2 and E-Ophtha). acceptable diagnostic performance of a DL system, developed
More recently, Gulshan and colleagues12 from Google AI and tested using the Singapore Integrated Diabetic Retinop-
Healthcare reported another DL system with excellent diag- athy Programme over a 5-year period, and 10 external data
nostic performance. The DL system was developed using 128 sets recruited from 6 different countries, including Singapore,
175 retinal images, graded between 3 and 7 times for DR China, Hong Kong, Mexico, USA and Australia. The DL system,
and DMO by a panel of 54 US licensed ophthalmologists and developed using the DL architecture VGG-19, was reported
ophthalmology residents between May and December 2015. The to have AUC, sensitivity and specificity of 0.936, 90.5% and
test set consisted of approximately 10 000 images retrieved from 91.6% in detecting referable DR. For vision-threatening DR,
two publicly available databases (EyePACS-1 and Messidor-2), the corresponding statistics were 0.958, 100% and 91.1%. The
168 Ting DSW, et al. Br J Ophthalmol 2019;103:167–175. doi:10.1136/bjophthalmol-2018-313173
 Review

Br J Ophthalmol: first published as 10.1136/bjophthalmol-2018-313173 on 25 October 2018. Downloaded from http://bjo.bmj.com/ on October 21, 2022 at India:BMJ-PG Sponsored.
AUC ranged from 0.889 to 0.983 for the 10 external data sets detection of any AMD. In this study, the authors used six convo-
(n=40 752 images). More recently, the DL system, developed lutional neural networks—AlexNet, GoogleNet, VGG, Incep-
by Abramoff et al,28 has obtained a US Food and Drug Admin- tion-V3, ResNet and Inception-ResNet-V2—to train different
istration approval for the diagnosis of DR. It was evaluated in models. Data augmentation was also used to increase the diver-
a prospective, although observational setting, achieving 87.2% sity of data set and to reduce the risk of overfitting. For the
sensitivity and 90.7% specificity.28 AREDS data set, all the photographs were captured as analogue
photographs and then digitised later. Whether this affects the DL
Age-related macular degeneration system’s performance remains uncertain. In addition, all three
AMD is a major cause of vision impairment in the elderly popu- abovementioned studies did not have any results for external
lation globally. The Age-Related Eye Disease Study (AREDS) validation on the individual DL systems.
classified AMD stages into none, early, intermediate and late
AMD.29 The American Academy of Ophthalmology recom-
mends that people with intermediate AMD should be at least DM, choroidal neovascularisation and other macular diseases
seen once every 2 years. It is projected that 288 million patients OCT has had a transformative effect on the management of
may have some forms of AMD by 2040,30 with approximately macular diseases, specifically neovascular AMD and DMO.
10% having intermediate AMD or worse.29 With the ageing OCT also provides a near-microscopic view of the retina in vivo
population, there is an urgent clinical need to have a robust DL with quick acquisition protocols revealing structural detail that
system to screen these patients for further evaluation in tertiary cannot be seen using other ophthalmic examination techniques.
eye care centres. Thus, the number of macular OCTs has grown from 4.3 million
Ting et al11 reported a clinically acceptable DL system diag- in 2012 to 6.4 million in 2016 in the US Medicare population
nostic performance in detecting referable AMD (table 1). alone, and will most likely continue to grow worldwide.31
Specifically, the DL system was trained and tested using 108 From a DL perspective, macular OCTs possess a number
558 retinal images from 38 189 patients. Fovea-centred images of attractive qualities as a modality for DL. First is the explo-
without macula segmentation were used in this study. Given that sive growth in the number of macular OCTs that are routinely
this was the DR screening population, there were relatively few collected around the world. This large number of OCTs is
patients with referable AMD. For the other two studies,17 18 DL required to train DL systems where having many training exam-
systems were developed using the AREDS data set, with a high ples can aid in the convergence of many-layered networks with
number of referable AMD (intermediate AMD or worse). Using millions of parameters. Second, macular OCTs have dense

Protected by copyright.
a fivefold cross-validation, Burlina et al17 reported a diagnostic three-dimensional structural information that is usually consis-
accuracy of between 88.4% and 91.6%, with an AUC of between tently captured. Unlike real-world images or even colour fundus
0.94 and 0.96. Unlike Ting et al,11 the authors presegmented the photographs, the field of view of the macula and the foveal
macula region prior to training and testing, with an 80/20 split fixation is usually consistent from one volume scan to another.
between the training and testing in each fold. In terms of the DL This lowers the complexity of the computer vision task signifi-
architecture, both AlexNet and OverFeat have been used, with cantly and allows networks to reach meaningful performance
AlexNet yielding a better performance. Using the same AREDS with smaller data sets. Third, OCTs provide structural detail
data set, Grassmann et al18 reported a sensitivity of 84.2% in the that is not easily visible using conventional imaging techniques

Table 2  Summary table for the different DL systems in the detection of retinal diseases using OCT
DL systems Year Disease OCT machines Test images CNN AUC Accuracy (%) Sensitivity (%) Specificity (%)
13 32
Lee et al 2017 Exudative AMD Spectralis 20 613 VGG-16 0.928 87.60 84.60 91.50
Trader et al33 2018 Exudative AMD Spectralis 100 Inception-V3 0.980 100 NA NA
Kermany et al34 2018 CNV Spectralis 1000 Inception-V3
DMO
Drusen
1. Multiclass comparison 0.999 96.50 97.80 97.40
2. Limited model 0.988 93.40 96.60 94.00
3. Binary model
CNV vs normal 1 100 100 100
DMO vs normal 0.999 98.20 96.80 99.60
Drusen vs normal 0.999 99 98 99.20
De Fauw et al43 2018 Urgent, semiurgent, Topcon 997 patients 1. Deep segmentation Urgent 94.5
routine and observation network using U-Net referral
only 0.992
Normal, CNV, macular Spectralis 116 patients 2. Deep classification Urgent 96.6
oedema, FTMH, PTMH, network using a custom 29 referral
CSR, VMT, GA, drusen, CNN layers with 5 pooling 0.999
ERM layers
The diagnostic performance is not comparable between the different DL systems given the different data sets used in the individual study. AUC for specific conditions: CNV
0.993; macular oedema 0.990; normal 0.995; FTMH 1.00; PTMH 0.999; CSR 0.995; VMT 0.980; GA 0.990; drusen 0.967; and ERM 0.966.
AMD, age-related macular degeneration; AUC, area under the receiver operating characteristic curve; CNN, convolutional neural network; CNV, choroidal neovascularisation; CSR,
central serous chorioretinopathy; DL, deep learning; DMO, diabetic macular oedema; ERM, epiretinal membrane; FTMH, full-thickness macula hole; GA, geographic atrophy; NA,
not available; OCT, optical coherence tomography; PTMH, partial thickness macula hole; VMT, vitreomacular traction.

Ting DSW, et al. Br J Ophthalmol 2019;103:167–175. doi:10.1136/bjophthalmol-2018-313173 169

 Review

Br J Ophthalmol: first published as 10.1136/bjophthalmol-2018-313173 on 25 October 2018. Downloaded from http://bjo.bmj.com/ on October 21, 2022 at India:BMJ-PG Sponsored.
and provide an avenue for uncovering novel biomarkers of the segmentation and classification tasks using a novel AI frame-
disease. work. In this approach, a segmentation network is first used to
One of the first applications of DL to macular OCTs was in delineate a range of 15 different retinal morphological features
automated classification of AMD. Approximately 100 000 OCT and OCT acquisition artefacts. The output of this network is
B-scans were used to train a DL classifier based on VGG-16 to then passed to a classification network which makes a referral
achieve an AUC of 0.97 (table 2).32 Few studies used a technique triage decision from four categories (urgent, semiurgent,
known as transfer learning, where a neural network is pretrained routine, observation) and classifies the presence of 10 different
on ImageNet and subsequently then trained on OCT B-scans for OCT pathologies (choroidal neovascularisation (CNV), macular
retinal disease classification.33–35 Of note, these initial studies oedema without CNV, drusen, geographic atrophy, epiretinal
involve the use of two-dimensional DL models trained on single membrane, vitreomacular traction, full-thickness macular hole,
OCT B-scans rather than three-dimensional models trained on partial thickness macular hole, central serous retinopathy and
OCT volumes. This may be a barrier to their potential clinical ‘normal’).43 Using this approach, the Moorfields-DeepMind
applicability. system reports a performance on par with experts for these clas-
DL has also had a transformative impact in boundary and sification tasks (although in a retrospective setting). Moreover,
feature-level segmentation using neural networks that have been the generation of an intermediate tissue representation by the
developed for semantic segmentation such as the U-Net.36 Specif- first, segmentation network means that the framework can be
ically, these networks have been trained to segment intraretinal generalised across OCT systems from multiple different vendors
fluid cysts and subretinal fluid on OCT B-scans.13 37 38 Deep without prohibitive requirements for retraining. In the near
convolutional networks surpassed traditional methods in the term, this DL system will be implemented in an existing real-
quality of segmentation of retinal anatomical boundaries.39–41 world clinical pathway—the rapid access ‘virtual’ clinics that are
Also similar approaches were used to segment en-face OCTA now widely used for triaging of macular disease in the UK.44 In
images to segment the foveal avascular zone.42 the longer term, the system could be used in triaging patients
More recently, DeepMind and the Moorfields Eye Hospital outside the hospital setting, particularly as OCT systems are
have combined the power of neural networks for both increasingly being adopted by optometrists in the community.45

Protected by copyright.

Figure 1  Archetype analysis with 16 visual field (VF) archetypes (ATs) that were derived from an unsupervised computer algorithm described by
Elze et al.49
170 Ting DSW, et al. Br J Ophthalmol 2019;103:167–175. doi:10.1136/bjophthalmol-2018-313173
 Review

Br J Ophthalmol: first published as 10.1136/bjophthalmol-2018-313173 on 25 October 2018. Downloaded from http://bjo.bmj.com/ on October 21, 2022 at India:BMJ-PG Sponsored.
Glaucoma optimised treatment regimens for the various forms of glau-
The global prevalence of glaucoma for people aged 40–80 is coma. Kazemian et al57 developed a clinical forecasting tool
3.4%, and by the year 2040 it is projected there will be approx- that uses tonometric and VF data to project disease trajecto-
imately 112 million affected individuals worldwide.46 Clinicians ries at different target IOPs. Further refinement of this tool that
and patients alike would welcome improvements in disease integrates other ophthalmic and non-ophthalmic data would be
detection, assessment of progressive structural and functional useful to establish target IOPs and the best strategies to achieve
damage, treatment optimisation so as to prevent visual disability, them on a case-by-case basis. Finally, it is documented that
and accurate long-term prognosis. patients with newly diagnosed glaucoma harbour fears of going
Glaucoma is an optic nerve disease categorised by excavation blind58; perhaps, the use of machine learning that incorporates
and erosion of the neuroretinal rim that clinically manifests itself genome-wide data, lifestyle behaviour and medical history into a
by increased optic nerve head (ONH) cupping. Yet, because forecasting algorithm will allow early prognostication regarding
the ONH area varies by fivefold, there is virtually no cup to the future risk of requiring invasive surgery or losing functional
disc ratio (CDR) that defines pathological cupping, hampering vision from glaucoma.
disease detection.47 Li et al16 and Ting et al11 trained computer As machine learning algorithms are revised, the practising
algorithms to detect the glaucoma-like disc, defined as a vertical ophthalmologist will have a host of tools available to diagnose
CDR of 0.7 and 0.8, respectively. Investigators have also applied glaucoma, detect disease progression and identify optimised
machine learning methods to distinguish glaucomatous nerve treatment strategies using a precision medicine approaches. In
fibre layer damage from normal scans on wide-angle OCTs an ideal future scenario, they may also have clinical forecasting
(9×12 mm).48 Future opportunities include training a neural tools that inform patients as to their overall prognosis and
network to identify the disc that would be associated with mani- expected clinical course with or without treatment.
fest visual field (VF) loss across the spectrum of disc size, as our
current treatment strategies are aligned with slowing disease
detection. Furthermore, DL could be used to detect progressive Retinopathy of prematurity
structural optic nerve changes in glaucoma. ROP is a leading cause of childhood blindness worldwide, with
In glaucoma, retinal ganglion cell axons atrophy in a confined an annual incidence of ROP-related blindness of 32 000 world-
space within the ONH and ophthalmologists typically rely on wide.59 The regional epidemiology of the disease varies based
low dimensional psychophysical data to detect the functional on a number of factors, including the number of preterm births,
consequences of that damage. The outputs from these tests neonatal mortality of preterm children and capacity to monitor

Protected by copyright.
typically provide reliability parameters, age-matched normative exposure to oxygen. ROP screening either directly via ophthal-
comparisons and summary global indices, but more detailed moscopic examination or telemedical evaluation using digital
analysis of this functional data is lacking. Elze et al49 developed fundus photography can identify the earliest signs of severe
an unsupervised computer program to analyse VF that recog- ROP, and with timely treatment can prevent most cases of blind-
nises clinically relevant VF loss patterns and assigns a weighting ness from ROP.60 61 Due to the high number of preterm births,
coefficient for each of them (figure 1). This method has proven reductions in neonatal mortality, and limited capacity for oxygen
useful in the detection of early VF loss from glaucoma.50 monitoring and ROP screening, the highest burden of blinding
Furthermore, a myriad of computer programs to detect VF ROP today is in low-income and middle-income countries.62
progression exist, ranging from assessment of global indices over There are two main barriers to effective implementation of
time to point-wise analyses, to sectoral VF analysis; however, ROP screening: (1) the diagnosis of ROP is subjective, with
these approaches are often not aligned with clinical ground truth significant interexaminer variability in the diagnosis leading to
nor with one another.51 52 Yousefi et al53 developed a machine- inconsistent application of evidence-based interventions63; and
based algorithm that detected VF progression earlier than these (2) there are too few trained examiners in many regions of the
conventional strategies. More machine learning algorithms that world.64 Telemedicine has emerged as a viable model to address
provide quantitative information about regional VF progression the latter problem, at least in regions where the cost of a fundus
can be expected in the future. camera is not prohibitive, by allowing a single physician to virtu-
Although intraocular pressure (IOP)-lowering has been ally examine infants over a large geographical area. However,
shown to be therapeutically effective in delaying glaucoma telemedicine itself does not solve the subjectivity problem in
progression, some demonstrated that disease progression is ROP diagnosis. Indeed, the acute-phase ROP study found nearly
still inevitable,54–56 suggesting that we have not arrived at 25% of telemedicine examinations by trained graders required

Table 3  The clinical and technical challenges in building and deploying deep learning (DL) techniques from ’bench to bedside’
Steps Potential challenges
1. Identification of training data 1. Patients’ consent and confidentiality issues.
sets 2. Varying standards and regulations between the different institutional review boards.
3. Small training data sets for rare disease (eg, ocular tumours) or common diseases that are not captured in routine (eg, cataracts).
2. Validation and testing data 1. Lack of sample size—not sufficiently powered.
sets 2. Lack of generalisability—not tested widely in different populations or on data collected from different devices.
3. Explainability of the results 1. Demonstration of the regions ‘deemed’ abnormal by DL.
2. Methods to generate heat maps—occlusion tests, class activation, integrated gradient method, soft attention map and so on.
4. Clinical deployment of DL 1. Recommendation of the potential clinical deployment sites.
Systems 2. Application of regulatory approval from health authorities (eg, US Food and Drug Administration, Europe CE marking and so on).
3. Conducting prospective clinical trials.
4. Medical rebate scheme and medicolegal requirement.
5. Ethical challenges.

Ting DSW, et al. Br J Ophthalmol 2019;103:167–175. doi:10.1136/bjophthalmol-2018-313173 171

 Review

Br J Ophthalmol: first published as 10.1136/bjophthalmol-2018-313173 on 25 October 2018. Downloaded from http://bjo.bmj.com/ on October 21, 2022 at India:BMJ-PG Sponsored.
adjudication because the graders disagreed on one of three Potential challenges
criteria for clinically significant ROP.65 Despite the high level of accuracy of the AI-based models in many
There have been a number of early attempts to use DL for of the diseases in ophthalmology, there are still many clinical and
automated diagnosis of ROP,19 66 which could potentially address technical challenges for clinical implementation and real-time
both implementation barriers for ROP screening. Most recently, deployment of these models in clinical practice (table 3). These
Brown et al19 reported the results of a fully automated DL system challenges could arise in different stages in both the research and
that could diagnose plus disease, the most important feature of clinical settings. First, many of the studies have used training data
severe ROP, with an AUC of 0.98 compared with a consensus sets from relatively homogeneous populations.12 14 15 AI training
reference standard diagnosis combining image-based diagnosis and testing using retinal images is often subject to numerous
and ophthalmoscopy (table 1). When directly compared with variabilities, including width of field, field of view, image magni-
the eight international experts in ROP diagnosis, the i-ROP DL fication, image quality and participant ethnicities. Diversifying
system agreed with the consensus diagnosis more frequently the data set, in terms of ethnicities, and image-capture hardware
than six out of eight experts. Subsequent work found that the could help to address this challenge.11
i-ROP DL system could also produce a severity score for ROP Another challenge in the development of AI models in ophthal-
that demonstrated promise for objective monitoring of disease mology has been the limited availability of large amounts of data
progression, regression and response to treatment.67 When for both the rare diseases (eg, ocular tumours) and for common
compared with the same set of 100 images ranked in order of diseases which are not imaged routinely in clinical practice such
disease severity by experts, the algorithm had 100% sensitivity as cataracts. Furthermore, there are diseases such as glaucoma
an 94% specificity in the detection of pre-plus or worse disease. and ROP where there will be disagreement and interobserver

Protected by copyright.

Figure 2  Some examples of heat maps showing the abnormal areas in the retina. (A) Severe non-proliferative diabetic retinopathy (NPDR); (B)
geographic atrophy in advanced age-related macular degeneration (AMD) on fundus photographs11; and (C) diabetic macular oedema on optical
coherence tomography.
172 Ting DSW, et al. Br J Ophthalmol 2019;103:167–175. doi:10.1136/bjophthalmol-2018-313173
 Review

Br J Ophthalmol: first published as 10.1136/bjophthalmol-2018-313173 on 25 October 2018. Downloaded from http://bjo.bmj.com/ on October 21, 2022 at India:BMJ-PG Sponsored.
variability in the definition of the disease phenotype. The algo- Third, large-scale adoption of AI in healthcare is still not on
rithm learns from what they are presented with. The software the horizon as clinicians and patients are still concerned about
is unlikely to produce accurate outcomes if the training set of AI and DL being ‘black-boxes’. In healthcare, it is not only
images given to the AI tool is too small or not representative the quantitative algorithmic performance, but the underlying
of real patient populations. More evidence on ways of getting features through which the algorithm classifies disease which
high-quality ground-truth labels is required for different imaging is important to improve physician acceptance. Generating heat
tools. Krause et al68 reported that adjudication grades by retina maps highlighting the regions of influence on the image which
specialists were a more rigorous reference standard, especially contributed to the algorithm conclusion may be a first step
to detect artefacts and missed microaneurysms in DR, than a (figure 2), although such maps are often challenging to interpret
majority decision and improved the algorithm performance. (what does it mean if a map highlights an area of vitreous on
Second, many AI groups have reported robust diagnostic an OCT of a patient with drusen?).69 They may also struggle to
performance for their DL systems, although some papers did not deal with negations (what would it mean to highlight the most
show how the power calculation was performed for the indepen- important part of an ophthalmic image that demonstrates that
dent data sets. A power calculation should take the following into there is no disease present?).70 71 An alternative approach has
consideration: the prevalence of the disease, type 1 and 2 errors, been used for the DL system developed by the Moorfields Eye
CIs, desired precision and so on. It is important to first preset Hospital and DeepMind—in this system, the generation of an
the desired operating threshold on the training set, followed by intermediate tissue representation by a segmentation network
analysis of performance metrics such as sensitivity and specificity is used to highlight for the clinician (and quantify) the rele-
on the test set to assess calibration of the algorithm. vant areas of retinal pathology (figure 3).43 It is also important

Protected by copyright.

Figure 3  A representative screenshot from the output of the Moorfields-DeepMind deep learning system for optical coherence tomography
segmentation and classification. In this case, the system correctly diagnoses a case of central serous retinopathy with secondary choroidal
neovascularisation and recommends urgent referral to an ophthalmologist. Through the creation of an intermediate tissue representation (seen here
as two-dimensional thickness maps for each morphological parameter), the system provides ’explainability’ for the ophthalmologist.
Ting DSW, et al. Br J Ophthalmol 2019;103:167–175. doi:10.1136/bjophthalmol-2018-313173 173
 Review

Br J Ophthalmol: first published as 10.1136/bjophthalmol-2018-313173 on 25 October 2018. Downloaded from http://bjo.bmj.com/ on October 21, 2022 at India:BMJ-PG Sponsored.
to highlight that ‘interpretability’ of DL systems may mean by the National Science Foundation (SCH-1622542, SCH-1622536, SCH-1622679)
different things to a healthcare professional than to a machine and by unrestricted departmental funding from Research to Prevent Blindness. PAK
is supported by a UK National Institute for Health Research (NIHR) Clinician Scientist
learning expert. Although it seems likely that interpretable Award (NIHR-CS--2014-12-023). The views expressed are those of the authors and
algorithms will be more readily accepted by ophthalmologists, not necessarily those of the NHS, the NIHR or the Department of Health.
future applied clinical research will be necessary to determine
Competing interests  DSWT and TYW are the coinventors of a deep learning
whether this is the case and whether it leads to tangible benefits system for retinal diseases. LP is a member of Google AI Healthcare. LRP is a non-
for patients in terms of clinical effectiveness. paid consultant for Visulytix. PAK is a consultant for DeepMind.
Lastly, the current AI screening systems for DR have been Patient consent  Not required.
developed and validated using two-dimensional images and lack
Provenance and peer review  Not commissioned; internally peer reviewed.
stereoscopic qualities, thus making identification of elevated
lesions like retinal tractions challenging. Incorporating the infor- Open access  This is an open access article distributed in accordance with the
Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which
mation from multimodal imaging in future AI algorithms may permits others to distribute, remix, adapt, build upon this work non-commercially,
potentially address this challenge. In addition, the medicolegal and license their derivative works on different terms, provided the original work is
aspects and the regulatory approvals vary in different countries properly cited, appropriate credit is given, any changes made indicated, and the use
and settings, and more work will be needed in these areas. An is non-commercial. See: http://​creativecommons.​org/​licenses/​by-​nc/​4.0
important challenge to the clinical adoption of AI-based tech-
nology is how the patients entrust clinical care to machines.
Keel et al72 evaluated the patient acceptability of AI-based DR
References
1. World Economic Forum, 2016. The fourth industrial revolution: what it means, how to
screening within endocrinology outpatient setting and reported respond. https://www.​weforum.​org/​agenda/​2016/​01/​the-​fourth-i​ ndustrial-​revolution-​
that 96% of participants were satisfied or very satisfied with the what-​it-​means-​and-​how-​to-​respond/ (accessed 18 Aug 2018).
automated screening model.72 However, in different populations 2. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436–44.
and settings, the patient’s acceptability for AI-based screening 3. Zhang X, Zou J, He K, et al. Accelerating very deep convolutional networks for
classification and detection. IEEE Trans Pattern Anal Mach Intell 2016;38:1943–55.
may vary and may pose challenge in its implementation. 4. Shin HC, Roth HR, Gao M, et al. Deep convolutional neural networks for computer-
aided detection: cnn architectures, dataset characteristics and transfer learning. IEEE
Trans Med Imaging 2016;35:1285–98.
Conclusions 5. Tompson J, Jain A, LeCun Y. Joint training of a convolutional network and a graphical
DL is the state-of-the-art AI machine learning technique that has model for human pose estimation. Advances in Neural Information Processing
revolutionised the AI field. For ophthalmology, DL has shown Systems 2014;27:1799–807.
6. Hinton G, Deng L, Yu D. Deep neural networks for acoustic modeling in speech

Protected by copyright.
clinically acceptable diagnostic performance in detecting many recognition. IEEE Signal Processing Magazine 2012;29:82–97.
retinal diseases, in particular DR and ROP. Future research is 7. Lakhani P, Sundaram B. Deep learning at chest radiography: automated classification
crucial in evaluating the clinical deployment and cost-effective- of pulmonary tuberculosis by using convolutional neural networks. Radiology
ness of different DL systems in the clinical practice. To improve 2017;284:574–82.
clinical acceptance of DL systems, it is important to unravel the 8. Ting DSW, Yi PH, Hui F. Clinical applicability of deep learning system in detecting
tuberculosis with chest radiography. Radiology 2018;286:729–31.
‘black-box’ nature of DL using existing and future methodolo- 9. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer
gies. Although there are challenges ahead, DL will likely impact with deep neural networks. Nature 2017;542:115–8.
on the practice of medicine and ophthalmology in the coming 10. Ehteshami Bejnordi B, Veta M, Johannes van Diest P, et al. Diagnostic assessment
decades. of deep learning algorithms for detection of lymph node metastases in women with
breast cancer. JAMA 2017;318:2199–210.
11. Ting DSW, Cheung CY, Lim G, et al. Development and validation of a deep learning
Author affiliations
1 system for diabetic retinopathy and related eye diseases using retinal images from
Singapore Eye Research Institute, Singapore National Eye Center, Duke-NUS Medical
multiethnic populations with diabetes. JAMA 2017;318:2211–23.
School, National University of Singapore, Singapore, Singapore
2 12. Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning
Department of Ophthalmology, Mt Sinai Hospital, New York City, New York, USA
3 algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA
Google AI Healthcare, Mountain View, California, USA
4 2016;316:2402–10.
Casey Eye Institute, Oregon Health and Science University, Portland, Oregon, USA
5 13. Lee CS, Tyring AJ, Deruyter NP, et al. Deep-learning based, automated segmentation
Department of Ophthalmology, University of Washington, School of Medicine,
of macular edema in optical coherence tomography. Biomed Opt Express
Seattle, Washington, USA
6 2017;8:3440–8.
Vitreo-retinal Department, Sankara Nethralaya, Chennai, Tamil Nadu, India
7 14. Abràmoff MD, Lou Y, Erginay A, et al. Improved automated detection of diabetic
Department of Ophthalmology, Lee Kong Chian School of Medicine, Nanyang
retinopathy on a publicly available dataset through integration of deep learning.
Technological University, Singapore, Singapore
8 Invest Ophthalmol Vis Sci 2016;57:5200–6.
Department of Clinical Pharmacology, Medical University of Vienna, Vienna, Austria
9 15. Gargeya R, Leng T. Automated identification of diabetic retinopathy using deep
Center for Medical Physics and Biomedical Engineering, Medical University of
Vienna, Vienna, Austria learning. Ophthalmology 2017;124:962–9.
10
Vitreo-retinal Service, Moorfields Eye Hospital, London, UK 16. Li Z, He Y, Keel S, et al. Efficacy of a deep learning system for detecting
glaucomatous optic neuropathy based on color fundus photographs. Ophthalmology
2018;125:1199–206.
Contributors  DSWT, LRP, LP, JPC, AYL, RR, GSWT, LS, PAK and TYW have all 17. Burlina PM, Joshi N, Pekala M, et al. Automated grading of age-related macular
contributed to manuscript drafting, literature review, critical appraisal and final degeneration from color fundus images using deep convolutional neural networks.
approval of the manuscript. JAMA Ophthalmol 2017;135:1170–6.
Funding  This project received funding from the National Medical Research Council 18. Grassmann F, Mengelkamp J, Brandl C, et al. A deep learning algorithm for prediction
(NMRC), Ministry of Health (MOH), Singapore National Health Innovation Center, of age-related eye disease study severity scale for age-related macular degeneration
Innovation to Develop Grant (NHIC-I2D-1409022), SingHealth Foundation Research from color fundus photography. Ophthalmology 2018;125:1410–20.
Grant (SHF/FG648S/2015), and the Tanoto Foundation, and unrestricted donations to 19. Brown JM, Campbell JP, Beers A, et al. Automated diagnosis of plus disease in
the Retina Division, Johns Hopkins University School of Medicine. For the Singapore retinopathy of prematurity using deep convolutional neural networks. JAMA
Epidemiology of Eye Diseases (SEED) study, we received funding from NMRC, MOH Ophthalmol 2018;136:803–10.
(grants 0796/2003, IRG07nov013, IRG09nov014, STaR/0003/2008 and STaR/2013; 20. Poplin R, Varadarajan AV, Blumer K, et al. Prediction of cardiovascular risk factors from
CG/SERI/2010) and Biomedical Research Council (grants 08/1/35/19/550 and retinal fundus photographs via deep learning. Nat Biomed Eng 2018;2:158–64.
09/1/35/19/616). The Singapore Integrated Diabetic Retinopathy Programme 21. Varadarajan AV, Poplin R, Blumer K, et al. Deep learning for predicting refractive error
(SiDRP) received funding from the MOH, Singapore (grants AIC/RPDD/SIDRP/SERI/ from retinal fundus images. Invest Ophthalmol Vis Sci 2018;59:2861–8.
FY2013/0018 and AIC/HPD/FY2016/0912). In USA, it is supported by the National 22. Yau JW, Rogers SL, Kawasaki R, et al. Global prevalence and major risk factors of
Institutes of Health (K12 EY027720, R01EY019474, P30EY10572, P41EB015896), diabetic retinopathy. Diabetes Care 2012;35:556–64.

174 Ting DSW, et al. Br J Ophthalmol 2019;103:167–175. doi:10.1136/bjophthalmol-2018-313173

 Review

Br J Ophthalmol: first published as 10.1136/bjophthalmol-2018-313173 on 25 October 2018. Downloaded from http://bjo.bmj.com/ on October 21, 2022 at India:BMJ-PG Sponsored.
23. Hutchinson A, McIntosh A, Peters J, et al. Effectiveness of screening and monitoring 48. Christopher M, Belghith A, Weinreb RN, et al. Retinal nerve fiber layer features
tests for diabetic retinopathy--a systematic review. Diabet Med 2000;17:495–506. identified by unsupervised machine learning on optical coherence tomography scans
24. Scanlon PH, Malhotra R, Greenwood RH, et al. Comparison of two reference predict glaucoma progression. Invest Ophthalmol Vis Sci 2018;59:2748–56.
standards in validating two field mydriatic digital photography as a method of 49. Elze T, Pasquale LR, Shen LQ, et al. Patterns of functional vision loss in glaucoma
screening for diabetic retinopathy. Br J Ophthalmol 2003;87:1258–63. determined with archetypal analysis. J R Soc Interface 2015;12.
25. Murray R, Metcalf SM, Lewis PM. Sustainingremote-area programs: retinal camera 50. Wang M, Pasquale LR, Shen LQ, et al. Reversal of glaucoma hemifield test results and
use by Aboriginal health workers and nurses in a Kimberley partnership. MedJ Aust visual field features in glaucoma. Ophthalmology 2018;125:352–60.
2005;182:520–3. 51. Tanna AP, Bandi JR, Budenz DL, et al. Interobserver agreement and intraobserver
26. Ting DS, Tay-Kearney ML, Constable I, et al. Retinal video recording a new way to reproducibility of the subjective determination of glaucomatous visual field
image and diagnose diabetic retinopathy. Ophthalmology 2011;118:1588–93. progression. Ophthalmology 2011;118:60–5.
27. Ting DS, Cheung GC, Wong TY. Diabetic retinopathy: global prevalence, major 52. Viswanathan AC, Crabb DP, McNaught AI, et al. Interobserver agreement on
risk factors, screening practices and public health challenges: a review. Clin Exp visual field progression in glaucoma: a comparison of methods. Br J Ophthalmol
Ophthalmol 2016;44:260–77. 2003;87:726–30.
28 Abramoff MD, Lavin PT, Birch M, et al. Pivotal trial of an autonomous AI-based 53. Yousefi S, Kiwaki T, Zheng Y, et al. Detection of longitudinal visual field progression in
diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ glaucoma using machine learning. Am J Ophthalmol 2018;193:71–9.
Digit Med 2018;1:1–8. 54. Kass MA, Heuer DK, Higginbotham EJ, et al. The Ocular Hypertension Treatment
29. Chew EY, Clemons TE, Agrón E, et al. Effect of omega-3 fatty acids, lutein/zeaxanthin, Study: a randomized trial determines that topical ocular hypotensive medication
or other nutrient supplementation on cognitive function: the areds2 randomized delays or prevents the onset of primary open-angle glaucoma. Arch Ophthalmol
clinical trial. JAMA 2015;314:791–801. 2002;120:701–13.
30. Wong WL, Su X, Li X, et al. Global prevalence of age-related macular degeneration 55. Heijl A, Leske MC, Bengtsson B, et al. Reduction of intraocular pressure and glaucoma
and disease burden projection for 2020 and 2040: a systematic review and meta- progression: results from the Early Manifest Glaucoma Trial. Arch Ophthalmol
analysis. Lancet Glob Health 2014;2:e106–16. 2002;120:1268–79.
31. Centers for Medicare & Medicaid Services, 2018. CMS medicare provider utilization 56. Comparison of glaucomatous progression between untreated patients with
and payment data. https://www.​cms.​gov/​Research-​Statistics-​Data-​and-​Systems/​ normal-tension glaucoma and patients with therapeutically reduced intraocular
Statistics-​Trends-​and-​Reports/​Medicare-​Provider-​Charge-​Data/​index.​html (accessed 4 pressures. Collaborative Normal-Tension Glaucoma Study Group. Am J Ophthalmol
Sep 2018). 1998;126:487–97.
32. Lee CS, Baughman DM, Lee AY. Deep learning is effective for classifying normal versus 57. Kazemian P, Lavieri MS, Van Oyen MP, et al. Personalized prediction of glaucoma
age-related macular degeneration OCT images. Ophthalmol Retina 2017;1:322–7. progression under different target intraocular pressure levels using filtered forecasting
33. Treder M, Lauermann JL, Eter N. Automated detection of exudative age-related methods. Ophthalmology 2018;125:569–77.
macular degeneration in spectral domain optical coherence tomography using deep 58. Lichter PR, Musch DC, Gillespie BW, et al. Interim clinical outcomes in the
learning. Graefes Arch Clin Exp Ophthalmol 2018;256:259–65. Collaborative Initial Glaucoma Treatment Study comparing initial treatment
34. Kermany DS, Goldbaum M, Cai W. Identifying medical diagnoses and treatable randomized to medications or surgery. Ophthalmology 2001;108:1943–53.
diseases by image-based deep learning. Cell 2018;172:1122–31. 59. Blencowe H, Moxon S, Gilbert C. Update on blindness due to retinopathy of
35. Ting DSW, Liu Y, Burlina P, et al. AI for medical imaging goes deep. Nat Med prematurity globally and in India. Indian Pediatr 2016;53(Suppl 2):S89–92.

Protected by copyright.
2018;24:539–40. 60. Robinson R, O’Keefe M. Cryotherapy for retinopathy of prematurity--a prospective
36. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image study. Br J Ophthalmol 1992;76:289–91.
segmentation. Medical Image Computing and Computer-Assisted Intervention – 61. Cryotherapy for Retinopathy of Prematurity Cooperative Group. Multicenter trial of
MICCAI, 2015. cryotherapy for retinopathy of prematurity: ophthalmological outcomes at 10 years.
37. Roy AG, Conjeti S, Karri SPK, et al. ReLayNet: retinal layer and fluid segmentation of Arch Ophthalmol 2001;119:1110–8.
macular optical coherence tomography using fully convolutional networks. Biomed 62. Gilbert C, Rahi J, Eckstein M, et al. Retinopathy of prematurity in middle-income
Opt Express 2017;8:3627–42. countries. Lancet 1997;350:12–14.
38. Venhuizen FG, van Ginneken B, Liefers B, et al. Deep learning approach for the 63. Fleck BW, Williams C, Juszczak E, et al. An international comparison of retinopathy of
detection and quantification of intraretinal cystoid fluid in multivendor optical prematurity grading performance within the Benefits of Oxygen Saturation Targeting II
coherence tomography. Biomed Opt Express 2018;9:1545–69. trials. Eye 2018;32:74–80.
39. Hamwood J, Alonso-Caneiro D, Read SA, et al. Effect of patch size and network 64. Campbell JP, Swan R, Jonas K, et al. Implementation and evaluation of a tele-
architecture on a convolutional neural network approach for automatic segmentation education system for the diagnosis of ophthalmic disease by international trainees.
of OCT retinal layers. Biomed Opt Express 2018;9:3049–66. AMIA Annu Symp Proc 2015;2015:366–75.
40. Fang L, Cunefare D, Wang C, et al. Automatic segmentation of nine retinal layer 65. Daniel E, Quinn GE, Hildebrand PL, et al. Validated system for centralized grading
boundaries in OCT images of non-exudative AMD patients using deep learning and of retinopathy of prematurity: telemedicine approaches to evaluating acute-phase
graph search. Biomed Opt Express 2017;8:2732–44. retinopathy of prematurity (e-ROP) study. JAMA Ophthalmol 2015;133:675–82.
41. Chen M, Wang J, Oguz I, et al. Automated segmentation of the choroid in EDI-OCT 66. Worrall D, Wilson CM, Brostow GJ. Automated retinopathy of prematurity case
images with retinal pathology using convolution neural networks. Fetal Infant detection with convolutional neural networks, 2016.
Ophthalmic Med Image Anal 2017;10554:177–84. 67. Brown JM, Campbell JP, Beers A. Fully automated disease severity assessment and
42. Prentašic P, Heisler M, Mammo Z, et al. Segmentation of the foveal microvasculature treatment monitoring in retinopathy of prematurity using deep learning. Proceedings
using deep learning networks. J Biomed Opt 2016;21:75008. Volume 10579, Medical Imaging 2018: Imaging Informatics for Healthcare, Research,
43. De Fauw J, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for and Applications, 2018.
diagnosis and referral in retinal disease. Nat Med 2018;24:1342–50. 68. Krause J, Gulshan V, Rahimy E, et al. Grader variability and the importance of
44. Buchan JC, Amoaku W, Barnes B, et al. How to defuse a demographic time bomb: the reference standards for evaluating machine learning models for diabetic retinopathy.
way forward? Eye 2017;31:1519–22. Ophthalmology 2018;125:1264–72.
45. 2018. OCT rollout in every specsavers announced. https://www.​aop.​org.​uk/​ot/​ 69. Ramanishka V, Das A, Zhang J, 2016. Top-down visual saliency guided by captions.
industry/​high-​street/​2017/​05/2​ 2/​oct-​rollout-​in-​every-​specsavers-a​ nnounced (accessed https://​arxiv.​org/a​ bs/​1612.​07360
4 Sep 2018). 70. Lam C, Yu C, Huang L, et al. Retinal lesion detection with deep learning using image
46. Tham YC, Li X, Wong TY, et al. Global prevalence of glaucoma and projections patches. Invest Ophthalmol Vis Sci 2018;59:590–6.
of glaucoma burden through 2040: a systematic review and meta-analysis. 71. Quellec G, Charrière K, Boudi Y, et al. Deep image mining for diabetic retinopathy
Ophthalmology 2014;121:2081–90. screening. Med Image Anal 2017;39:178–93.
47. Jonas JB, Gusek GC, Guggenmoos-Holzmann I, et al. Size of the optic nerve scleral 72. Keel S, Lee PY, Scheetz J, et al. Feasibility and patient acceptability of a novel artificial
canal and comparison with intravital determination of optic disc dimensions. Graefes intelligence-based screening model for diabetic retinopathy at endocrinology
Arch Clin Exp Ophthalmol 1988;226:213–5. outpatient services: a pilot study. Sci Rep 2018;8:4330.

Ting DSW, et al. Br J Ophthalmol 2019;103:167–175. doi:10.1136/bjophthalmol-2018-313173 175