1082 Natl Sci Rev, 2019, Vol. 6, No.

6 PERSPECTIVES

Downloaded from https://academic.oup.com/nsr/article/6/6/1082/5484863 by guest on 17 August 2021

GEOSCIENCES

Deep neural network for remote-sensing image interpretation: status

and perspectives
Jiayi Li1 , Xin Huang 1,2,∗ and Jianya Gong1

Deep neural networks (DNNs) refer to technologies for the large-scale geospa- representative remote-sensing bench-
end-to-end mappings (i.e. from data to tial information mining [1]. While some mark datasets, further investigation
information) by stacking a large num- progress has been made, research in on task-driven deep learning (i.e. the
ber of filters learned from massive sam- deep-learning-based remote-sensing im- integration of deep learning and remote-
ples. By courtesy of the comprehensive age interpretation is still in its infancy, sensing physical mechanisms) and the
Earth observation platforms and conve- mainly subject to insufficient annota- efforts towards promoting the practical-
nient data access, remote-sensing prac- tion samples, high complexity of the ity of the networks should be considered
titioners are dealing with very large and model, and lack of in-depth integra- in the agenda. In this context, this paper
ever-growing data volumes, which call for tion between deep learning and remote aims to summarize the developments
fast and transferrable machine-learning sensing. Construction of diverse and of DNNs for remote-sensing image
PERSPECTIVES Li et al. 1083

Practicability
N
0 3 6 12 miles

RS Earth Observation

2005
<100 100-200 200-300
300-400 400-500 >500 (104 kWh)
Energy consumption estimation

2017

Un-used

Downloaded from https://academic.oup.com/nsr/article/6/6/1082/5484863 by guest on 17 August 2021

Residential Trees Infrastructure
Commercial Waters Grasslands
Industrial Breeds Croplands
Land cover monitoring

Reliability
Operational use
RS practitioners Urban planning

Data

LiDAR and
aerial Technique
optical
CNN
image [7] Unfolding

RNN

AE U-Net

Training Real sample

set
D
Pixel-wise
Airplane Chaparral Grass Sea
Bare-soil Court Mobile-home Ship label G Fake sample
Buildings Dock Pavement Tanks GAN
Cars Field Sand Trees Water

Volume Heterogeneity Supervised learning Advanced learning

Variety Difficulty of labeling Pre-training Novel RS technique

Figure 1. Deep learning for remote-sensing (RS) image interpretation, from the perspectives of data, technology and practicality.

interpretation from the aspects of data, deep learning, the public, scientific constructed in the fields of computation
technology and practicality (Fig. 1). and industrial communities are paying vision or natural language processing, to
constant attention to its technological remote-sensing image classification, ob-
advances [2]. Remotely sensed im- ject detection, spatio-temporal analysis,
DEVELOPMENT OF DEEP agery is a typical image data source etc. More recently, with the in-depth
LEARNING FOR REMOTE with periodic Earth observation. The development of the deep-learning tech-
SENSING overwhelming advantages of DNNs have niques, current research has focused on
Since 2013, neural networks with a core been presented in many remote-sensing the use of pre-trained models, in spite
of deep learning have entered the third applications. In the earliest stages, of their limitations for generalization in
climax of artificial intelligence (AI) remote-sensing researchers tended to complex remote-sensing applications. In
research. With the explosive growth of apply the existing networks that were this context, as an area closely related to
1084 Natl Sci Rev, 2019, Vol. 6, No. 6 PERSPECTIVES

AI, remote-sensing image interpretation true-color small parcels (e.g. 256 × (iv) Difficulty of labeling: The expen-
is facing both great opportunities and 256 feet per parcel in [6]) for tens of sive label annotation, including in-
challenges. Please refer to [3] for glos- classes, which are mainly used for image situ information, ground survey,
saries of the terms in this paper. retrieval. The second category involves matched data and expert knowl-
the extremely-high-spatial-resolution edge, limits the size of remote-
STATUS AND PERSPECTIVES aerial images with centimeter ground res- sensing datasets. Moreover, un-
olution [7]. On the basis of these efforts, der the circumstances of large-
Data several studies train models from scratch scale sample size, the label noise
Deep learning is essentially a process and start to promote the development of from time inconsistency between
of learning big data using large-scale DNNs from the perspective of remote remote-sensing observation and
computing power. Large-scale datasets sensing [8]. However, these remote- labeling is more severe.
in diverse areas not only bring up public sensing benchmark datasets are still in In short, reliable, large-volume, di-

Downloaded from https://academic.oup.com/nsr/article/6/6/1082/5484863 by guest on 17 August 2021

benchmarks for evaluating the scalable their infancy, owing to the following verse and representative benchmark data
and diverse DNN works, but also deficiencies: are vital for developing deep learning in
improve the visibility, availability and the field of remote sensing. There is an ur-
feasibility of the DNN models. Com- (i) Data volume: There is a limited gent need for building multi-modal and
pared with natural images, obtaining amount of information in these multi-platform datasets with sufficient
priori remote-sensing samples is more ex- datasets in spite of the large data land-cover diversity, terrestrial types and
pensive, which requires extensive labor- volume they have. In contrast to various imaging conditions.
intensive expert analyses (including field natural image datasets that can
survey and visual inspection). Thus, the successfully train large-scale para-
mainstream approach [4] is to use a metric networks with thousands Technology
small number of labeled remote-sensing of layers, current remote-sensing
samples to fine-tune the existing DNN datasets are at high risk of overfit- Deep learning designs hierarchical
models, which have been pre-trained on ting the models when interpreted architecture by stacking several blocks
large-scale data from other fields. How- with such deep architecture, since composed of filters (or layers) with spe-
ever, a recent study reveals that the use of they only measure a small area cific functions to capture the information
the ‘pre-training and fine-tuning’ strategy of the surface coverage and lack in the images. Convolution-based and
does not necessarily improve the final tar- general representative ability. recurrence-based operators, embedded
get task accuracy [5], even if the network (ii) Data heterogeneity: In addition to in the convolutional neuro network
is pre-trained on a similar task. Besides, a few instances that provide Light (CNN) and recurrent neuro network
there are numerous differences between Detection And Ranging (LiDAR) (RNN), respectively, are two state-of-
natural images and remote-sensing data. and very-high-spatial-resolution the-art filters for remote-sensing images.
First, in contrast to the natural images multispectral data, in most cases, For CNN, the convolutional layer en-
with three bands (i.e. red, green, blue), each dataset is collected from a ables the network to integrate the multi-
remote-sensing images have many more single data source. The hetero- scale spatial measurements, which have
spectral channels, such as from the geneity of the remote-sensing potential for exploring the contextual
ultraviolet to the microwave spectrum. datasets with diverse modalities information of remote-sensing images.
Therefore, normally, the quality (e.g. (e.g. multi-sensor, multi-temporal With regard to RNN, the directed graph
signal-noise ratio) of remote-sensing and multi-resolution) and plat- along a temporal sequence formed by the
images is much lower. Meanwhile, the forms (e.g. constellation) can connections between recurrent neuron
spatial dependence and spectral inter- challenge their collaboration and ensures its advantage in processing
dependence of remote-sensing images transferability. Moreover, the multi-temporal remote-sensing images.
violate the basic assumption of the current datasets tend to focus However, currently, scientists tend to
natural image datasets, namely identical on urban areas, which limit their construct their networks by borrowing
and independent data distribution. The generalization when applied to or fusing the existing ones that originate
imaging geometries, the differences of monitoring natural scenes. from other fields. From the viewpoint of
multiple sensors, as well as the imaging (iii) Data variety: Most of the cur- a learning paradigm, DNN-based works
conditions in multiple time series can rent image samples depict a small in the field of remote sensing can be
further challenge the applicability of the and simple region clipped from divided into four categories:
network trained by natural images for a large image scene, but ignore (i) Supervised learning approaches
remote-sensing interpretation. various remote-sensing imaging trained from scratch. With the aid
To this end, remote-sensing com- conditions (e.g. shadow, clouds), of current remote-sensing datasets
munities are starting to establish their making it difficult to adapt to the and the typical filters, some studies
own datasets. The first category of these real and complex applications of design and train a small task-
datasets are composed of hundreds of Earth observation. specific architecture. As the current
PERSPECTIVES Li et al. 1085

applications (e.g. land-use classifi- in recent years. It comprises Practicability

cation) of deep learning in remote two networks that compete with
Successful stories are still lacking for
sensing only involve small-scale each other in a zero-sum game
deep learning in the field of remote
pilot projects, these small net- framework—that is, the generative
sensing. Although DNNs have reached
works can avoid the dilemmas of network generates candidates while
superiority to some degree, they are
the large cost and risk in training the discriminative network evaluates
far from ‘standardization and commer-
the existing ‘large-scale’ networks them. This adversarial framework
cialization’. The development status of
with high redundancy and over- can drive both sub-networks to
DNNs cannot fully meet the needs of
parametrization [8]. Nevertheless, improve their performances until
diverse and complex remote-sensing
considering the rapid development the fakes are indistinguishable from
applications (e.g. territorial, agriculture,
of remote-sensing data-acquisition the genuine articles, and hence
atmosphere, urban). In addition to the
capabilities and the great demand for can overcome the difficulty of
above situations, the following issues

Downloaded from https://academic.oup.com/nsr/article/6/6/1082/5484863 by guest on 17 August 2021

diverse and complex remote-sensing inaccurate parameter estimation in
that restrict practicability should be
applications, such a learning ap- the conventional generative model.
addressed:
proach suffers from the limitation of Meta-learning, the latest progress in
labeled sample size when designing transfer learning, intends to rapidly (i) In terms of reliability, the geospa-
large-scale DNNs. learn new skills or adapt to new tasks tial information interpreted from
(ii) Pre-training and fine-tuning ap- with a few training examples and remote-sensing data should be
proaches. When dealing with meta knowledge. With these ad- robust with confidence estimates. In
complex remote-sensing interpreta- vanced learning paradigms, deep and this context, efforts in the following
tion tasks, most of the current work task-driven architecture [10] cus- two aspects (but not limited to)
either directly uses or fine-tunes tomized for remote-sensing image can be conducted: (a) conducting
the existing network pre-trained by interpretation is worthy of further uncertainty analysis to promise
large-scale data from other fields [4]. exploration. the confidence of the information
The key to the feasibility of these (iv) Novel technologies developed by extracted by DNNs and (b) investi-
learning approaches is based on the the remote-sensing community. To gating the functions of DNN layers
transplantability of these pre-trained address the specific problems in to facilitate the direct use of these
networks in interpreting data with the remote-sensing field on the layers as a feature extractor.
spatial/temporal hierarchy, such as basis of newly developed DNN (ii) In terms of operational use, on the
the similarity of the data or task. technologies, some novel filters one hand, the large-scale data
Owing to the low demand for sample have been designed in the most volume and the dense time-series
size and convenient implementa- recent studies. For instance, the information-extraction tasks call for
tion, this kind of transfer learning is blocks, including convolution light-weight networks with high-
the most commonly used strategy layers, activation function and throughput processing and real-
in remote sensing. However, the pooling layers, are extended to the time/quasi-real-time technology.
gap between the data from remote complex domain to represent the On the other hand, the concurrent
sensing and other fields, as well as amplitude and phase information monitoring tasks that come from
the high specificity of the deeper of synthetic aperture radar imagery the multi-functional satellites (e.g.
layers in the existing pre-trained [11]. Focusing on a certain kind inversion of multiple land surface
networks, inevitably restricts the of remote-sensing data source (e.g. parameters from MODIS (i.e.,
performance of such pre-training radar, hyperspectral and LiDAR), Moderate Resolution Imaging Spec-
approaches. Considering the col- a potential research direction is to troradiometer) data) also require
laborative development of dataset develop a physics-based model for further development of multi-task
construction and DNN-model analysing the data structure and DNN models.
design, it seems necessary to rethink understanding the physical process (iii) In terms of the remote-sensing prac-
and revise the ‘pre-training and fine- of remote sensing. Moreover, with titioners, in the era of AI, the bar
tuning’ paradigm for remote-sensing the development of automatic for building a DNN model is being
applications. machine learning (e.g. Google’s lowered. Several tools and platforms
(iii) Advanced learning. Recently, the AutoML), it is also worthy of fur- (e.g. Google’s AutoML) for deep
semi/un/weak supervised DNN ther research to construct more learning have been available. In
algorithms have been attracting flexible and specific architecture to this context, instead of learning
increasing attention, due to their low fuse various remote-sensing data DNN from scratch, the users can
cost for collecting remote-sensing sources and promote deep learning conveniently construct their deep
samples [9]. In particular, generative from the perspective of remote networks by only focusing on the
adversarial networks is a promising sensing. input samples. However, more
unsupervised algorithm developed remote-sensing-oriented DNN
1086 Natl Sci Rev, 2019, Vol. 6, No. 6 PERSPECTIVES

examples and libraries with open REFERENCES Geographic Information Systems (ACM GIS), 2010,
licenses are necessary and in-depth 270–9.
integration between remote sensing 1. Qin D, Ding Y and Xiao C et al. Natl Sci Rev 2018; 7. ISPRS 2d semantic labeling dataset. http://
and the DNN model is also needed. 5: 255–68. www2.isprs.org/commissions/comm3/wg4/
2. 10 breakthrough technologies 2018. MIT Tech semantic-labeling.html (14 February 2019, date
Rev. http://www.technologyreview.com/lists/ last accessed).
FUNDING technologies/2018/ (20 May 2019, date last 8. Li Y, Zhang Y and Huang X et al. IEEE Trans Geosci
This work was supported by the National Natu- accessed). Remote Sens 2018; 56: 6521–36.
ral Science Foundation of China (41771360 and 3. Glossaries of the terms. https://www. 9. Li Y, Huang X and Liu H. Photogramm Eng Remote
41842035), the National Program for Support of
researchgate.net/publication/332093559 Sens 2017; 83: 567–79.
Top-notch Young Professionals and the Hubei
Definition of terms 1 (20 May 2019, date last 10. Xu Z and Sun J. Natl Sci Rev 2018; 5: 22–4.
Provincial Natural Science Foundation of China
accessed). 11. Zhang Z, Wang H and Xu F et al. IEEE Trans Geosci
(2017CFA029).

Downloaded from https://academic.oup.com/nsr/article/6/6/1082/5484863 by guest on 17 August 2021

4. Amirkolaee HA and Arefi H. ISPRS J Photogramm Remote Sens 2017; 55: 7177–88.
Jiayi Li1 , Xin Huang 1,2,∗ and Jianya Gong1
Remote Sens 2019; 149: 50–66.
1 School of Remote Sensing and Information
5. He K, Girshick R and Dollár P. 2018,
Engineering, Wuhan University, China
arXiv:1811.08883.
2 State Key Laboratory of Information Engineering
6. Yang Y and Newsam S. Bag-of-visual-words and National Science Review
in Surveying, Mapping and Remote Sensing,
spatial extensions for land-use classification. 6: 1082–1086, 2019
Wuhan University, China
In: ACM Proceedings of the 18th SIGSPATIAL doi: 10.1093/nsr/nwz058
∗ Corresponding author.
International Conference on Advances in Advance access publication 2 May 2019
E-mail: [email protected]